- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 4/6] container/webhook: start
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 container/webhook/start diff --git a/container/webhook/start b/container/webhook/start new file mode 100755 index 0000000..50cbf20 --- /dev/null +++ b/container/webhook/start @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "webhook" + +cmd = %W[ + docker run + --restart=always + --name webhook + -d + -p 11301:11301 + node:webhook +] + +cmd += ['sh', '-c', 'node webhook.js'] +system(*cmd) -- 2.23.0

2 2

[PATCH v2 lkp-tests] libmicro: add PKGBUILD for libmicro
by Bai Jing 05 Nov '20

05 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..61cd32f4 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://github.com/rzezeski/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

4 7

[PATCH lkp-tests] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 05 Nov '20

05 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } PKGBUILD-1.1 contains only the changed part. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..dc8471b9 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$1" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

4 4

[PATCH lkp-tests] add sysbench-threads-git.yaml
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] job.yaml transfer required parameters to testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/sysbench-threads-git.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 jobs/sysbench-threads-git.yaml diff --git a/jobs/sysbench-threads-git.yaml b/jobs/sysbench-threads-git.yaml new file mode 100644 index 00000000..6b0f2c00 --- /dev/null +++ b/jobs/sysbench-threads-git.yaml @@ -0,0 +1,21 @@ +suite: sysbench-threads-git +category: benchmark + +nr_threads: +- 2 +- 4 +- 8 +- 16 +- 32 +- 64 +- 128 +- 256 +- 512 +- 1024 +- 2048 +- 4096 +- 8192 + +sysbench-threads-git: + thread_yields: 100 + thread_locks: 2 -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 05 Nov '20

05 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..6d24d8d9 --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,36 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + if [ -x /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + else + echo "The hackbench executable file cannot be found." && exit 1 + fi + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

2 3

[PATCH lkp-tests] add sysbench-threads-git.yaml
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] job.yaml transfer required parameters to testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/sysbench-threads-git.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 jobs/sysbench-threads-git.yaml diff --git a/jobs/sysbench-threads-git.yaml b/jobs/sysbench-threads-git.yaml new file mode 100644 index 00000000..fe7552a4 --- /dev/null +++ b/jobs/sysbench-threads-git.yaml @@ -0,0 +1,21 @@ +suite: sysbench-threads-git +category: benchmark + +nr_threads: +- 2 +- 4 +- 8 +- 16 +- 32 +- 64 +- 128 +- 256 +- 512 +- 1024 +- 2048 +- 4096 +- 8192 + +sysbench-threads: + thread_yields: 100 + thread_locks: 2 -- 2.23.0

2 1

[PATCH v2 compass-ci 03/10] container/mail-robot: parse-apply-account-email.rb
by Luan Shengde 05 Nov '20

05 Nov '20

parse email for apply account check out commit url check commit url exists check base url in upstream-repos check commit available hub: gitee.com clone the repo and check the commit exists hub: non-gitee.com check the feedback of curl command to check the commit url available check out pub key extract the attachment of the pub_key Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/parse-apply-account-email.rb | 135 ++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100755 container/mail-robot/parse-apply-account-email.rb diff --git a/container/mail-robot/parse-apply-account-email.rb b/container/mail-robot/parse-apply-account-email.rb new file mode 100755 index 0000000..1325c8f --- /dev/null +++ b/container/mail-robot/parse-apply-account-email.rb @@ -0,0 +1,135 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check whether there is a commit url and ssh pub_key in the email +# apply account on junper server +# the entry point is parse_commit_url_pub_key +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + end + + def parse_commit_url_pub_key + commit_url = check_out_commit_url + pub_key = check_out_pub_key + + return commit_url, pub_key + end + + def send_error_email(error_message) + send_error_email_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'error_message' => error_message + } + send_err_mail = SendApplyAccountEmail.new(send_error_email_info) + send_err_mail.send_error_email + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = 'No matched commit url found' + send_error_email(error_message) + + raise error_message + end + + def check_out_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + check_base_url_in_upstream_repos('/c/upstream-repos', base_url) + check_commit_available(url, base_url) + + return url + end + + def check_base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = 'The url is not in upstream-repo list' + send_error_email(error_message) + + raise error_message + end + + def check_commit_available(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + check_gitee_commit(url, base_url) + else + check_non_gitee_commit(url) + end + end + + def check_non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_email + + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_gitee_commit(url, base_url) + repo_dir = url.split('/')[-3] + repo_url = [base_url, 'git'].join('.') + commit_id = url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_email + + FileUtils.rm_rf repo_dir + + check_gitee_commit_exist(email_index) + end + + def check_gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_out_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + check_pub_key(pub_key) + + return pub_key + end + + def check_pub_key(pub_key) + return unless pub_key.nil? + + error_message = 'No pub_key found' + send_error_email(error_message) + + raise error_message + end +end -- 2.23.0

4 7

[PATCH v2 compass-ci 02/10] container/mail-robot: apply-account.rb
by Luan Shengde 05 Nov '20

05 Nov '20

check to assign account invoke ParseApplyAccountEmail to parse commit url and pubkey generate uuid and invoke AssignJumperAccount to apply account store account info to es send uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/apply-account.rb | 65 +++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100755 container/mail-robot/apply-account.rb diff --git a/container/mail-robot/apply-account.rb b/container/mail-robot/apply-account.rb new file mode 100755 index 0000000..9a8d706 --- /dev/null +++ b/container/mail-robot/apply-account.rb @@ -0,0 +1,65 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '../../lib/es_client' + +# assign uuid/account +class AssignAccount + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + end + + def check_to_send_account + my_commit_url, my_ssh_pubkey = check_out_commit_url_pub_key + my_uuid, acct_infos = check_to_apply_account(my_ssh_pubkey) + + my_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_commit_url' => my_commit_url, + 'my_login_name' => acct_infos['my_login_name'], + 'my_uuid' => my_uuid + } + store_account_info(my_info) + my_info['acct_infos'] = acct_infos + send_my_uuid(my_info) + end + + def check_out_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + my_commit_url, my_ssh_pubkey = parse_apply_account_email.parse_commit_url_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def check_to_apply_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + apply_account_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_uuid' => my_uuid, + 'my_ssh_pubkey' => my_ssh_pubkey + } + apply_account = AssignJumperAccount.new(apply_account_info) + acct_infos = apply_account.apply_jumper_account + + return my_uuid, acct_infos + end + + def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) + end + + def send_my_uuid(my_info) + send_uuid = SendApplyAccountEmail.new(my_info) + send_uuid.send_uuid_email + end +end -- 2.23.0

5 8

[PATCH v3 lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] Adapting sysbench-threads needs old version to user Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..41ed9f46 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=1 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() +{ + cd "$srcdir/sysbench" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() +{ + cd "$srcdir/sysbench" + make DESTDIR=$pkgdir install +} -- 2.23.0

5 4

[PATCH v4 lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] Adapting sysbench-threads needs old version to user Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..41ed9f46 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=1 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() +{ + cd "$srcdir/sysbench" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() +{ + cd "$srcdir/sysbench" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 05 Nov '20

05 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..799d3c4d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,36 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + if [ -f /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + elif [ -f /usr/local/bin/hackbench ]; then + hackbench_path=/usr/local/bin/hackbench + else + echo "The hackbench executable file cannot be found." + fi + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

3 3

[PATCH v2 compass-ci] sbin/es-repo-mapping.sh: create script that generates "repo" mapping
by Zhang Yuhang 05 Nov '20

05 Nov '20

[why] We need a script to create "repo" mapping in es. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-repo-mapping.sh | 47 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 sbin/es-repo-mapping.sh diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh new file mode 100755 index 0000000..3a3930e --- /dev/null +++ b/sbin/es-repo-mapping.sh @@ -0,0 +1,47 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Check if "repo" index already exists. +status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +[ $status_code -eq 200 ] && echo '"repo" index already exists. Exit.' && exit + +# Create "repo" index. +echo 'Start to create "repo" index.' +curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +{ + "mappings": { + "_doc": { + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" + } + } + } + } +}' +[ $? -ne 0 ] && echo 'Failed to create "repo" index.' -- 2.23.0

2 2

[PATCH v5 lkp-tests 2/2] tests/iozone: modify the way of parsing parameter
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] modify the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b6a1e206 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write") args+=" -i 0" + ;; + "read") args+=" -i 1" + ;; + "rand_rw") args+=" -i 2" + ;; + "backwards") args+=" -i 3" + ;; + "record") args+=" -i 4" + ;; + "stride") args+=" -i 5" + ;; + "fwrite") args+=" -i 6" + ;; + "fread") args+=" -i 7" + ;; + "rand_mix") args+=" -i 8" + ;; + "pwrite") args+=" -i 9" + ;; + "pread") args+=" -i 10" + ;; + "pwritev") args+=" -i 11" + ;; + "preadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

3 3

[PATCH compass-ci 6/6] container/webhook: add scripts for webhook
by Li Yuanchao 05 Nov '20

05 Nov '20

webhook.js will listen to webhook message from 'gitee.com' or 'github.com' if catch 'push' message, call push_hook.rb to send message to git-mirror through message queue Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/root/push_hook.rb | 15 +++++++++++++++ container/webhook/root/webhook.js | 22 ++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/webhook/root/push_hook.rb create mode 100755 container/webhook/root/webhook.js diff --git a/container/webhook/root/push_hook.rb b/container/webhook/root/push_hook.rb new file mode 100755 index 0000000..e467ccd --- /dev/null +++ b/container/webhook/root/push_hook.rb @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +connection = Bunny.new('amqp://172.17.0.1:5672') +connection.start +channel = connection.create_channel + +queue = channel.queue('web_hook') +message = ARGV[0] +queue.publish(message) +connection.close diff --git a/container/webhook/root/webhook.js b/container/webhook/root/webhook.js new file mode 100755 index 0000000..a98576c --- /dev/null +++ b/container/webhook/root/webhook.js @@ -0,0 +1,22 @@ +const http = require('http') +const spawn = require('child_process').spawn +const createHandler = require('git-webhook-handler') +// if users want to use webhook, they need to config webhook on their code warehouse('gitee.com' or 'github.com'), +// and use url: http://183.134.196.212:11301/webhook, secret: webhook@git. +const handler = createHandler({ path: '/webhook', secret: 'webhook@git' }) + +handler.on('error', function(err){ + console.error('Error:', err.message) +}) + +handler.on('push', function(event){ + console.log(event.payload.repository.url) + spawn('ruby', ['push_hook.rb', event.payload.repository.url]) +}) + +http.createServer(function(req, res){ + handler(req, res, function(err){ + res.statusCode = 404 + res.end('no such location') + }) +}).listen(11301) -- 2.23.0

2 1

[PATCH compass-ci 3/6] container/webhook: build
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/webhook/build diff --git a/container/webhook/build b/container/webhook/build new file mode 100755 index 0000000..bb5c0a7 --- /dev/null +++ b/container/webhook/build @@ -0,0 +1,6 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t node:webhook . + -- 2.23.0

2 1

[PATCH compass-ci 2/6] container/webhook: Dockerfile
by Li Yuanchao 05 Nov '20

05 Nov '20

add container webhook Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/webhook/Dockerfile diff --git a/container/webhook/Dockerfile b/container/webhook/Dockerfile new file mode 100644 index 0000000..a9f0356 --- /dev/null +++ b/container/webhook/Dockerfile @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM node:10.21.0-alpine + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +ADD root / + +RUN apk update && \ + apk upgrade && \ + apk add --no-cache git && \ + apk add npm ruby-dev + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + +RUN npm install git-webhook-handler &&\ + gem install bunny + -- 2.23.0

3 2

[PATCH lkp-tests 2/2] nginx: add nginx PKGBUILD
by Wei Jihui 05 Nov '20

05 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx/PKGBUILD | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 pkg/nginx/PKGBUILD diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx/PKGBUILD new file mode 100644 index 00000000..b437c4b1 --- /dev/null +++ b/pkg/nginx/PKGBUILD @@ -0,0 +1,24 @@ +pkgname=nginx +pkgver=1.14 +pkgrel=2 +arch=('i386' 'x86_64' 'aarch64') +url="https://nginx.org" +license=('BSD') +depends=(pcre zlib openssl gcc) +source=("https://nginx.org/download/nginx-${pkgver}.#{pkgrel}.tar.gz") +sha256sums=('SKIP') + +build() +{ + benchmark_path="${pkgdir}/lkp/benchmarks/${pkgname}" + mkdir -p "${benchmark_path}" + cd "${srcdir}/nginx-${pkgver}.#{pkgrel}" + ./configure --with-http_ssl_module --prefix="${benchmark_path}" + make +} + +package() +{ + cd "$srcdir/$pkgname-$pkgver" + make install +} -- 2.23.0

3 2

[PATCH lkp-tests] fio-setup-basic: keep format when change create_task
by Xu Xijian 05 Nov '20

05 Nov '20

[why] When add time_based to create_task, add '\' first, keep this format with others in context. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- setup/fio-setup-basic | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..5b0d0fce 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -117,7 +117,7 @@ ioscheduler=$ioscheduler" fi if parse_bool -q "$time_based"; then - create_task=" + create_task="\ $create_task time_based " -- 2.23.0

1 0

[PATCH compass-ci 5/6] container/webhook: start-depends
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start-depends | 2 ++ 1 file changed, 2 insertions(+) create mode 100755 container/webhook/start-depends diff --git a/container/webhook/start-depends b/container/webhook/start-depends new file mode 100755 index 0000000..38fc696 --- /dev/null +++ b/container/webhook/start-depends @@ -0,0 +1,2 @@ +rabbitmq +es -- 2.23.0

1 0

[PATCH compass-ci] container/master-fluentd: fix buffer warn
by Wu Zhende 05 Nov '20

05 Nov '20

[warn] [warn]: suppressed same stacktrace failed to emit fluentd's log event tag="fluent.warn" event={"message"=>"failed to write data into buffer by buffer overflow"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" [warn]: failed to write data into buffer by buffer overflow [How] Set num_threads=10 Flush the cache with 10 thread pools， prevents logs from piling up Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index a57d4cd..62c0319 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -44,6 +44,7 @@ port "#{ENV['LOGGING_ES_PORT']}" suppress_type_name true flush_interval 1s + num_threads 10 index_name ${tag} ssl_verify false log_es_400_reason true -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] compare values by each metrics
by Lu Weitao 05 Nov '20

05 Nov '20

compare values by each metrics based on groups matrices, and format compare result as echart data_set background: To support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: compare_values by each metrics ---> format/show results Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 107 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 078028a..ebe1cd2 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -399,8 +399,115 @@ def compare_group_matrices(group_matrices, options) result_str end +# input: groups_matrices +# { +# group_key_1 => { +# dimension_1 => matrix_1, (openeuler 20.03) +# dimension_2 => matrix_2, (openeuler 20.09) +# dimension_3 => matrix_3, (centos 7.6) +# }, +# group_key_2 => {...} +# } +# +# output: compare_metrics_values +# { +# group_key_1 => { +# metric_1 => { +# 'average' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'standard_deviation' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'change' => { +# 'dimension_2 vs dimension_1' => xxx +# 'dimension_3 vs dimension_1' => xxx +# 'dimension_3 vs dimension_2' => xxx +# } +# }, +# metric_2 => {...} +# } +# } +def compare_metrics_values(groups_matrices) + metrics_compare_values = {} + groups_matrices.each do |group_key, dimensions| + metrics_compare_values[group_key] = get_metric_values(dimensions) + end + metrics_compare_values +end + +def get_metric_values(dimensions) + metrics_values = {} + dimensions.each do |dim, matrix| + matrix.each do |metric, values| + assign_metric_values(metrics_values, dim, metric, values) + end + end + assign_metric_change(metrics_values) + metrics_values +end + +def assign_metric_values(metrics_values, dim, metric, values) + metrics_values[metric] ||= {} + metrics_values[metric]['average'] ||= {} + metrics_values[metric]['standard_deviation'] ||= {} + metric_value = get_values(values, true) + metrics_values[metric]['average'][dim] = metric_value[:average] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] +end + +def assign_metric_change(metrics_values) + metrics_values.each do |metric, values| + metrics_values[metric]['change'] = {} + next if values['average'].size < 2 + + dimension_list = values['average'].keys + change_titles = get_change_titles(dimension_list) + change_titles.each do |base_dimension, challenge_dimension| + change = get_compare_value(values['average'][base_dimension], values['average'][challenge_dimension], true) + values['change'] = { "#{challenge_dimension} vs #{base_dimension}" => change } + end + end +end + +# input: dimension_list +# eg: ['openeuler 20.03', 'debian sid', 'centos 7.6'] +# output: Array(base_dimension: String, challenge_dimension: String) +# [ +# ['openeuler 20.03', 'debian sid'], +# ['openeuler 20.03', 'centos 7.6'], +# ['debian sid', 'centos 7.6'] +# ] +def get_change_titles(dimension_list) + dims = [] + dimension_list_size = dimension_list.size + (1..dimension_list_size - 1).each do |i| + (i..dimension_list_size - 1).each do |j| + dims << [dimension_list[i - 1], dimension_list[j]] + end + end + dims +end + +def show_chart_compare_result(chart_result) + print JSON.pretty_generate(chart_result) +end + # Format Fields +def format_for_chart(metrics_compare_values, template_params) + chart_result = {} + chart_result['title'] = template_params['title'] || nil + chart_result['unit'] = template_params['unit'] || nil + chart_result['x_name'] = template_params['x_params'].join('|') || nil + chart_result['tables'] = metrics_compare_values + chart_result +end + def format_fails_runs(fails, runs) fails_width = (SUB_LONG_COLUMN_WIDTH * FAILS_PROPORTION).to_i runs_width = SUB_LONG_COLUMN_WIDTH - fails_width - 1 -- 2.23.0

2 1

[PATCH compass-ci] sbin/es-repo-mapping.sh: create script that generates "repo" mapping
by Zhang Yuhang 05 Nov '20

05 Nov '20

[why] We need a script to create "repo" mapping in es. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-repo-mapping.sh | 78 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 sbin/es-repo-mapping.sh diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh new file mode 100755 index 0000000..ce0ad74 --- /dev/null +++ b/sbin/es-repo-mapping.sh @@ -0,0 +1,78 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Determine whether curl is installed. If not, install curl. +if ! [ -x "$(command -v curl)" ] +then + echo "curl not exists, try to install." + if [ -x "$(command -v apk)" ] + then + apk add install -y curl + elif [ -x "$(command -v yum)" ] + then + yum install -y curl + elif [ -x "$(command -v apt-get)" ] + then + apt-get install -y curl + fi +else + echo "curl has install." +fi + +# Determine whether curl is installed successfully +if [ $? -ne 0 ] +then + echo "curl install failed, exit." + exit +fi + +# Determine whether repo index has created +status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) + +if [ $status_code -eq 200 ] +then + echo "repo index has create, exit." +else + echo "repo index not exists, begin create index." + curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' + { + "mappings": { + "_doc": { + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" + } + } + } + } + }' + if [ $? -ne 0 ] + then + echo "create repo index failed." + fi +fi -- 2.23.0

5 8

[PATCH v5 lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..026b97e2 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write,read,rand_rw block_size: - 64k -- 2.23.0

2 1

[PATCH compass-ci] container: delete invalid start depends fluentd
by Liu Yinsi 05 Nov '20

05 Nov '20

fluentd renamed master-fluentd, but scheduler and extract-stats have no dependency with master-fluentd now, so remove it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/extract-stats/start-depends | 1 - container/scheduler/start-depends | 1 - 2 files changed, 2 deletions(-) diff --git a/container/extract-stats/start-depends b/container/extract-stats/start-depends index 4a016e6..efbe5a2 100755 --- a/container/extract-stats/start-depends +++ b/container/extract-stats/start-depends @@ -1,2 +1 @@ taskqueue -fluentd diff --git a/container/scheduler/start-depends b/container/scheduler/start-depends index 3e28dc1..4a17b46 100755 --- a/container/scheduler/start-depends +++ b/container/scheduler/start-depends @@ -1,4 +1,3 @@ redis -fluentd es taskqueue -- 2.23.0

2 2

[PATCH lkp-tests 1/2] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] add allow_mounted_write when write mode Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..f90963aa 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -116,6 +116,12 @@ $create_task ioscheduler=$ioscheduler" fi +if [[ "$rw" =~ "write" ]]; then + create_task="\ +$create_task +allow_mounted_write=1" +fi + if parse_bool -q "$time_based"; then create_task=" $create_task -- 2.23.0

2 1

2024

2023

2022

2021

2020

Compass-ci