November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/8] lib/mail-robot.rb
by Luan Shengde 03 Nov '20

03 Nov '20

mail robot for applying account monitor mailbox check new added email file in new email directory handle new email file check to apply account only if email's subject match 'apply account' Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/mail-robot.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..9437078 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + begin + handle_new_email(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + check_to_apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def check_to_apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_dir("#{ENV['MAILDIR']}/new", "#{ENV['MAILDIR']}/cur") -- 2.23.0

2 1

[PATCH lkp-tests] bin/lkp-setup-rootfs: change the script that called when packaging
by Li Ping 02 Nov '20

02 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 2890601e..680027e6 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -187,14 +187,14 @@ define_files_pkg_pack() ( local benchmark=${1#pkg/} export benchmark=${benchmark%/PKGBUILD} - $LKP_SRC/tests/makepkg + $LKP_SRC/tests/cci-makepkg ) } # There may be $program-dev or $program-dev-dev in the # "$LKP_SRC/distro/depends" directory, # when the value passed by $define_files is "distro/depends/$program-dev", -# you don't know which script in "makepkg" or "pack-deps" to execute, +# you don't know which script in "cci-makepkg" or "cci-depends" to execute, # so it needs to be handled sparately. # Save the result to # "/srv/initrd/deps/$os/$os_arch/$os_version/$deps_benchmark.cgz", @@ -208,7 +208,7 @@ define_files_deps_pack() [ $suffix == "dev" ] && [ -f "$LKP_SRC/pkg/$pkg_benchmark/PKGBUILD" ] && { ( export benchmark=$pkg_benchmark - $LKP_SRC/tests/makepkg + $LKP_SRC/tests/cci-makepkg ) return 0 } @@ -216,7 +216,7 @@ define_files_deps_pack() ( export benchmark=$deps_benchmark export pack_arch=$os_arch - $LKP_SRC/tests/pack-deps + $LKP_SRC/tests/cci-depends ) } -- 2.23.0

1 0

[PATCH compass-ci 8/8] doc/mail-robot.md
by Luan Shengde 02 Nov '20

02 Nov '20

documentation for mail-robot service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..d8ac188 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url +attahcment + ssh pub_key + - id_rsa.pub + +## steps overview + +1. monitor + - the monitor will listen to the mailbox for new email files + handle new email file: + check to send account + email subject: apply account + +2. assign account + - check email + check commit url + check whether there is a standard commit url + check whether the repo's url in upstream-repos + check whether the commit url valid + if the hub name is gitee.com + # authentication is needed qwhen use curl to check the url + clone the repo and check the commit + if the hub name is not gitee.com + check the comit with curl + + check pub_key + check whether there is a ssh pub_key in the email file + + it will return failure email if any check end with failure + + - generate uuid + - apply account + build user_info + - my_email + - my_name + - my_uuid + - lab + - pub_key + execute command to apply account on jumper and return the account info + curl -XGET "${JUMPER_HOST}:${JUMPER_PORT}/assign_account" -d '${user_info}' + write my_email, my_name, my_uuid, lab to the defaule config file: + + - store apply infos + apply_infos: + - my_email + - my_name + - my_commit_url + - my_login_name + - my_uuid + invoke es to store the apply infos + + - send uuid + build send uuid email message with the apply_infos + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

1 0

[PATCH compass-ci 7/8] container/mail-robot: start
by Luan Shengde 02 Nov '20

02 Nov '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/start | 41 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..671eaf2 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT + MAILDIR + ROBOT_TMP_REPOS_DIR +] + +defaults = relevant_defaults(names) + +defaults['CCI_SRC'] = '/c/compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']}:rw + -w #{defaults['CCI_SRC']}/container/mail-robot + --log-driver json-file + mail-robot +] + +cmd += ['ruby', 'run.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci 6/8] container/mail-robot: build
by Luan Shengde 02 Nov '20

02 Nov '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH compass-ci 5/8] container/mail-robot: Dockerfile
by Luan Shengde 02 Nov '20

02 Nov '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..5eea701 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport + +RUN mkdir -p /repos_dir -- 2.23.0

1 0

[PATCH compass-ci 4/8] container/mail-robot: run.rb
by Luan Shengde 02 Nov '20

02 Nov '20

invoke apply-account-component, apply-account.rb, mail-rebot.rb to run the service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/run.rb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 container/mail-robot/run.rb diff --git a/container/mail-robot/run.rb b/container/mail-robot/run.rb new file mode 100755 index 0000000..b432d7d --- /dev/null +++ b/container/mail-robot/run.rb @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'apply-account-component' +require_relative 'apply-account' +require_relative '../../lib/mail-robot' -- 2.23.0

1 0

[PATCH lkp-tests 2/2] tests/iozone: modify parameter parse process
by Lu Kaiyi 02 Nov '20

02 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool need change parse process. [how] modify parse process Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 50 ++++++++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..19c246f1 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,36 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + [ "$ele" == "write_rewrite" ] && args+=" -i 0" + [ "$ele" == "read_reread" ] && args+=" -i 1" + [ "$ele" == "random_read_write" ] && args+=" -i 2" + [ "$ele" == "read_backwards" ] && args+=" -i 3" + [ "$ele" == "rewrite_record" ] && args+=" -i 4" + [ "$ele" == "stride_read" ] && args+=" -i 5" + [ "$ele" == "fwrite_refwrite" ] && args+=" -i 6" + [ "$ele" == "fread_refread" ] && args+=" -i 7" + [ "$ele" == "random_mix" ] && args+=" -i 8" + [ "$ele" == "pwrite_repwrite" ] && args+=" -i 9" + [ "$ele" == "pread_repread" ] && args+=" -i 10" + [ "$ele" == "pwritev_repwritev" ] && args+=" -i 11" + [ "$ele" == "preadv_repreadv" ] && args+=" -i 12" + done else args+=" -a" fi -- 2.23.0

1 0

[PATCH lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 02 Nov '20

02 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..f9ab305f 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write_rewrite,read_reread,random_read_write block_size: - 64k -- 2.23.0

1 0

[PATCH lkp-tests 2/2] tests/iozone: modify parameter parse process
by Lu Kaiyi 02 Nov '20

02 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool need change parse process. [how] modify parse process Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 50 ++++++++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..19c246f1 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,36 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + [ "$ele" == "write_rewrite" ] && args+=" -i 0" + [ "$ele" == "read_reread" ] && args+=" -i 1" + [ "$ele" == "random_read_write" ] && args+=" -i 2" + [ "$ele" == "read_backwards" ] && args+=" -i 3" + [ "$ele" == "rewrite_record" ] && args+=" -i 4" + [ "$ele" == "stride_read" ] && args+=" -i 5" + [ "$ele" == "fwrite_refwrite" ] && args+=" -i 6" + [ "$ele" == "fread_refread" ] && args+=" -i 7" + [ "$ele" == "random_mix" ] && args+=" -i 8" + [ "$ele" == "pwrite_repwrite" ] && args+=" -i 9" + [ "$ele" == "pread_repread" ] && args+=" -i 10" + [ "$ele" == "pwritev_repwritev" ] && args+=" -i 11" + [ "$ele" == "preadv_repreadv" ] && args+=" -i 12" + done else args+=" -a" fi -- 2.23.0

1 0