- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/4] assign-account: generate rsa keys
by Luan Shengde 11 Nov '20

11 Nov '20

generate rsa private/public key and return public key on jumper server [why] register the public key on testbox enable user one click to login in the testbox [how] generate rsa private/public key for user Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 32 ++++++++++++-------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..b5a2388 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -94,30 +94,36 @@ class AccountStorage end def setup_jumper_account_info - account_info = read_account_info - jumper_info = read_jumper_info + login_name, password = read_account_info + jumper_host, jumper_port = read_jumper_info pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? - login_name = account_info[0] - password = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + password = 'Use pub_key to login' unless pub_key.nil? + + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + my_jumper_pubkey = generate_ssh_key(login_name) if @data['my_jumper_pubkey'].eql? true jumper_account_info = { 'my_login_name' => login_name, 'my_password' => password, - 'jumper_host' => jumper_info[0].chomp, - 'jumper_port' => jumper_info[1].chomp + 'jumper_host' => jumper_host, + 'jumper_port' => jumper_port, + 'my_jumper_pubkey' => my_jumper_pubkey } - config_authorized_key(login_name, pub_key) unless pub_key.nil? - config_default_yaml(login_name) - return jumper_account_info end + def generate_ssh_key(login_name) + ssh_dir = File.join('/home/', login_name, '.ssh') + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + hostname = File.read('/etc/hostname') + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@#{hostname}") + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + File.read("/home/#{login_name}/.ssh/id_rsa.pub") + end + def config_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') FileUtils.mkdir_p default_yaml_dir -- 2.23.0

1 0

[PATCH lab-z9] hosts: fix testbox included yaml error [Why] submit job with some physical machine's textbox will error: Included yaml file not found: 'hosts/taishan200-2280-2s48p-256g' (RuntimeError)
by Wei Jihui 11 Nov '20

11 Nov '20

[How] lkp-tests just included files that find $LKP_SRC and testbox's directory, so delete hosts is ok. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a103 | 2 +- hosts/taishan200-2280-2s48p-256g--a104 | 2 +- hosts/taishan200-2280-2s48p-256g--a106 | 2 +- hosts/taishan200-2280-2s48p-256g--a107 | 2 +- hosts/taishan200-2280-2s48p-512g--a105 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 index 304c6fc..9fcc718 100644 --- a/hosts/taishan200-2280-2s48p-256g--a103 +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bdc69 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 index a8a2789..41cdbb0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a104 +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8892431 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 index ad587b3..95b2f05 100644 --- a/hosts/taishan200-2280-2s48p-256g--a106 +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883bf99 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 index fba1937..31b8dc6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a107 +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917c39 diff --git a/hosts/taishan200-2280-2s48p-512g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 index 096b5b0..66a0aca 100644 --- a/hosts/taishan200-2280-2s48p-512g--a105 +++ b/hosts/taishan200-2280-2s48p-512g--a105 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-512g +<<: taishan200-2280-2s48p-512g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891945d -- 2.23.0

2 1

[PATCH v4 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh export_top_envexport_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..370c7899 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

3 2

[PATCH v4 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..f829199 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

2 2

[PATCH v6 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fbe9c432 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$file" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

3 4

[PATCH lkp-tests] depends: add depends to sysbench-mysql
by Zhang Yu 11 Nov '20

11 Nov '20

Use source code install mysql, so no need mysql-server here. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index e9585216..cfbc4518 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,2 +1,2 @@ -mysql-server -libmysqlclient-dev +libnuma1 +xfsprogs -- 2.23.0

1 0

[PATCH v1 compass-ci] splitting the cifs services
by Xiao Shenwei 11 Nov '20

11 Nov '20

[why] our os-cifs container will mount three dir, /srv/os /srv/initrd /srv/result it's may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container usage: if you want mount these floder, you can use these command server: /srv/os client: mount -t cifs -o guest //ip/os /tmp/os server: /srv/initrd client: mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd server: /srv/result client: mount -t cifs -o guest,port=447 //ip/result /tmp/result --- container/os-cifs/start | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index 6cc08c6..e20f887 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -8,18 +8,40 @@ lsmod | grep -q "^cifs\s" || { sudo modprobe cifs } -docker_rm samba +docker_rm os-cifs +docker_rm initrd-cifs +docker_rm result-cifs -cmd=( +os_cifs_cmd=( docker run -dt -p 445:445 + -v /etc/localtime:/etc/localtime:ro -v /srv/os:/srv/os + --name os-cifs + --restart=always + alpine/samba +) + +initrd_cifs_cmd=( + docker run -dt + -p 446:445 -v /etc/localtime:/etc/localtime:ro -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/samba +) + +result_cifs_cmd=( + docker run -dt + -p 447:445 + -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result - --name os-cifs + -name result-cifs --restart=always alpine/samba ) -"${cmd[@]}" +"${os_cifs_cmd[@]}" +"${initrd_cifs_cmd[@]}" +"${result_cifs_cmd[@]}" -- 2.23.0

2 2

[PATCH lkp-tests] job2sh: remove atomic job test
by Wu Fengguang 11 Nov '20

11 Nov '20

Make it easier to do simple demo. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/job2sh | 6 ------ sbin/job2sh.cr | 6 ------ 2 files changed, 12 deletions(-) diff --git a/sbin/job2sh b/sbin/job2sh index 4fb57453..9ada3cfa 100755 --- a/sbin/job2sh +++ b/sbin/job2sh @@ -38,12 +38,6 @@ begin job = Job2sh.new job.load(ARGV[0]) job.expand_params - - unless job.atomic_job? - log_error "Looks #{ARGV[0]} isn't a atomic jobfile, only atomic jobfile is supported" - log_error "Please run 'lkp split-job #{ARGV[0]}' first" - exit 1 - end rescue Job::ParamError => e log_error "Abandon job: #{e.message}" exit 1 diff --git a/sbin/job2sh.cr b/sbin/job2sh.cr index ffcd3359..3a921bd6 100644 --- a/sbin/job2sh.cr +++ b/sbin/job2sh.cr @@ -28,12 +28,6 @@ begin job = Job2sh.new job.load(ARGV[0]) job.expand_params - - unless job.atomic_job? - log_error "Looks #{ARGV[0]} isn't a atomic jobfile, only atomic jobfile is supported" - log_error "Please run 'lkp split-job #{ARGV[0]}' first" - exit(1) - end rescue e : Job::ParamError log_error "Abandon job: #{e.message}" exit(1) -- 2.23.0

1 0

[PATCH lkp-tests] tests: fix lmbench3 tool adapt to aarch64 framework
by Bai Jing 11 Nov '20

11 Nov '20

error: cat: results/x84_64-linux-gnu/taishan200-2280-2s48p-256g--a28.0 No such file or directory Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/lmbench3 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/lmbench3 b/tests/lmbench3 index 8570a35a..dc4d157d 100755 --- a/tests/lmbench3 +++ b/tests/lmbench3 @@ -138,7 +138,7 @@ run_lmbench3_development() eval "$test=yes" - sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/x86_64-linux-gnu/lmbench + sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/"${os_arch}"-linux-gnu/lmbench log_eval " ( echo $nr_threads @@ -177,7 +177,7 @@ run_lmbench3_development() show_result() { - cat results/x86_64-linux-gnu/$testbox.0 || { + cat results/"${os_arch}"-linux-gnu/$testbox.0 || { echo "no test result found" exit -1 } -- 2.23.0

1 0

[PATCH v4 compass-ci] container: fix failed to build archlinux images in x86_64 machine
by Liu Yinsi 11 Nov '20

11 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error:404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 because archlinux mirror not support x86_64 machine. [how] root/etc/pacman.d/mirrorlist => root/aarch64/etc/pacman.d/mirrorlist root/x86_64/etc/pacman.d/mirrorlist use ARCH=$(arch) to choose mirrorlist according to different system architecture. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/Dockerfile | 6 +++++- container/archlinux/build | 2 +- .../archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist | 1 + 4 files changed, 7 insertions(+), 2 deletions(-) rename container/archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) create mode 100644 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/Dockerfile b/container/archlinux/Dockerfile index c0f05d3..1f80ae0 100644 --- a/container/archlinux/Dockerfile +++ b/container/archlinux/Dockerfile @@ -5,7 +5,11 @@ FROM lopsided/archlinux MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> -COPY root / +ARG ARCH + +COPY root/$ARCH / + RUN chmod 755 /etc + RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..9749489 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t archlinux:testbed . +docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/archlinux/root/etc/pacman.d/mirrorlist b/container/archlinux/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/archlinux/root/etc/pacman.d/mirrorlist rename to container/archlinux/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..556fac8 --- /dev/null +++ b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch -- 2.23.0

1 0