- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/3] [multi-docker] obtain the memory,cpu from the hostname
by Xiao Shenwei 10 Nov '20

10 Nov '20

hostname before: dc-1g--$HOSTNAME now : dc-1g.$HOSTNAME Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 7234dfa..9edad62 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -13,7 +13,7 @@ if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then else tbox_group=$hostname fi -host=${tbox_group%%--*} +host=${tbox_group%.*} create_yaml_variables "$LKP_SRC/hosts/${host}" -- 2.23.0

2 2

[PATCH v3 compass-ci 03/11] mail-robot: check to assign account
by Luan Shengde 10 Nov '20

10 Nov '20

check to assign account invoke ParseApplyAccountEmail parse commit url parse pubkey invoke AssignJumperAccount generate uuid apply account store account info to es send email send success email send fail email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 73 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..880de97 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,73 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(message) + email_message = if message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 02/11] mail-robot: listen new email directory
by Luan Shengde 10 Nov '20

10 Nov '20

monitor mailbox for new email handle new email file check to apply account only if email's subject match 'apply account' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..168e64b --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +# require_relative 'send-apply-account-email' +# require_relative 'send-uuid-email' +# require_relative 'send-error-email' +# require_relative 'apply-jumper-account' +# require_relative 'parse-apply-account-email' +# require_relative 'assign-account' + +# MAILDIR = '/srv/cci/Maildir/.compass-ci' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

2 4

[PATCH compass-ci] kernel_version.md: add default vmlinuz in Related files example
by Xu Xijian 10 Nov '20

10 Nov '20

[why] Add default vmlinuz in Related files example, it is a softlink. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md index f6a5f4d..23c132a 100644 --- a/doc/job/kernel_version.md +++ b/doc/job/kernel_version.md @@ -34,6 +34,7 @@ Related files: ├── modules-4.19.90-2003.cgz ├── modules.cgz -> modules-4.19.90-2003.cgz ├── vmlinuz-4.19.90-2003 +├── vmlinuz -> vmlinuz-4.19.90-2003 Usage example: - submit iperf.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

2 2

[PATCH lkp-tests] bin/lkp-setup-rootfs: fix mount error
by Li Ping 10 Nov '20

10 Nov '20

[problem]: submit -a cci-makepkg.yaml testbox=taishan200-2280-2s64p-256g--a42 cat /srv/result/cci-makepkg/taishan200-2280-2s64p-256g--a42/2020-11-05/z9.146070/output mount error(16): Device or resource busy Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) Failed to run mount /lkp/lkp/src/lib/debug.sh:12: die /lkp/lkp/src/tests/cci-makepkg:37: main [why]: submit with the option '-a' will call the define_files function and add test-user change files to job's define_files. Then define_files_aoto_pack will be triggered. When submit -a cci-makepkg.yaml, $LKP_SRC/tests/cci-makepkg will run twice, which results in mounting twice and causes mount error. Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 680027e6..83029f09 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -251,7 +251,9 @@ chmod_root_ssh while true; do set_tbox_wtmp 'running' . $job_script define_files - define_files_auto_pack $define_files + [ "$suite" != "cci-makepkg" ] && [ "$suite" != "cci-depends" ] && { + define_files_auto_pack $define_files + } if [ "$os_mount" != "initramfs" ]; then wget_install_cgz $initrd_deps wget_install_cgz $initrd_pkg -- 2.23.0

3 3

[PATCH lkp-tests] tests/*: fix double mount problem
by Wang Yong 10 Nov '20

10 Nov '20

[Why] when use submit -a and run cci-makepkg/cci-depends job, lkp will call cci-makepkg/cci-depends first and destination dir will mount twice when job is running the second time [How] check mountpoint before mount destination dir Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 ++ tests/cci-makepkg | 2 ++ 2 files changed, 4 insertions(+) diff --git a/tests/cci-depends b/tests/cci-depends index efa9a025b..4b885dc0c 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -7,6 +7,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh +. $LKP_SRC/lib/mount.sh [ -n "$benchmark" ] || die "benchmark is empty" [ -n "$os_mount" ] || die "os_mount is empty" @@ -28,6 +29,7 @@ pack_arch=$os_arch . $LKP_SRC/distro/$DISTRO +is_mount_point ${DEPS_MNT} && umount ${DEPS_MNT} mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$DEPS_MNT $DEPS_MNT || die "Failed to run mount" umask 002 diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe2..a105f1895 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -13,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh . $LKP_SRC/lib/env.sh +. $LKP_SRC/lib/mount.sh [ -n "$benchmark" ] || die "benchmark is empty" [ -n "$os_mount" ] || die "os_mount is empty" @@ -34,6 +35,7 @@ pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" [ -n "$LKP_SERVER" ] && { + is_mount_point ${PKG_MNT} && umount ${PKG_MNT} mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } -- 2.23.0

2 1

[PATCH compass-ci 2/3] [multi-docker] support queues parameter
by Xiao Shenwei 09 Nov '20

09 Nov '20

register host2queues and requested specified queues Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 47 ++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 4e6bc2f..8e489c6 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,15 +11,26 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT +] +defaults = relevant_defaults(names) +SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = defaults['SCHED_PORT'] || 3000 + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 - "http://#{host}:#{port}/boot.container/hostname/#{hostname}" + "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" +end + +def set_host2queues(hostname, queues) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/set_host2queues?host=#{hostname}&queues=#{queues}'" + system cmd +end + +def del_host2queues(hostname) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/del_host2queues?host=#{hostname}'" + system cmd end def parse_response(url) @@ -67,7 +78,7 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end -def run(hostname, load_path, hash) +def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( @@ -77,24 +88,26 @@ def run(hostname, load_path, hash) clean_dir(load_path) end -def main(hostname) +def main(hostname, queues) + set_host2queues(hostname, queues) url = get_url hostname puts url hash = parse_response url - return if hash.nil? + return del_host2queues(hostname) if hash.nil? load_path = build_load_path(hostname) load_initrds(load_path, hash) - run(hostname, load_path, hash) + start_container(hostname, load_path, hash) + del_host2queues(hostname) end -def loop_main(hostname) +def loop_main(hostname, queues) loop do begin - main(hostname) + main(hostname, queues) rescue StandardError => e puts e.backtrace - # if an exception happend, request the next time after 30 seconds + # if an exception occurs, request the next time after 30 seconds sleep 25 ensure sleep 5 @@ -109,11 +122,11 @@ def save_pid(pids) f.close end -def multi_docker(hostname, nr_container) +def multi_docker(hostname, nr_container, queues) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop_main("#{hostname}-#{i}") + loop_main("#{hostname}-#{i}", queues) end pids << pid end -- 2.23.0

1 0

[PATCH v2 compass-ci] container: fix failed to build kibana images
by Liu Yinsi 09 Nov '20

09 Nov '20

[why] when build kibana images in x86_machine, error message: [root@localhost kibana]# ./build Sending build context to Docker daemon 5.12kB Step 1/3 : FROM gagara/kibana-oss-arm64:7.6.2 7.6.2: Pulling from gagara/kibana-oss-arm64 38163f410fa0: Pull complete 69a4d016f221: Pull complete 95e6c6e7c9ca: Pull complete d13f429dd982: Pull complete 508bb3330fb2: Pull complete 9634e726f1b6: Pull complete 9c26c37850c8: Pull complete 0d0ad8467060: Pull complete 940f92726f8b: Pull complete Digest: sha256:541632b7e9780a007f8a8be82ac8853ddcebcb04a596c00500b73f77eacfbd16 Status: Downloaded newer image for gagara/kibana-oss-arm64:7.6.2 ---> f482a0472f78 Step 2/3 : MAINTAINER Wu Zhende <wuzhende666(a)163.com> ---> Running in cfa86d8ce976 Removing intermediate container cfa86d8ce976 ---> 3be6c5f24d4b Step 3/3 : RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml ---> Running in ff455f66df8b standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/kibana/Dockerfile | 4 +++- container/kibana/build | 8 +++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 35802fe..6e0dba0 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -1,7 +1,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/kibana-oss-arm64:7.6.2 +ARG BASE_IMAGE + +FROM BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 diff --git a/container/kibana/build b/container/kibana/build index a7e4717..52d5a2a 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,4 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -system 'docker build -t kibana:7.6.2 .' +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', + 'x86_64' => 'kibana:7.6.2' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." -- 2.23.0

1 0

[PATCH v1 lkp-tests] tests: adapt the special version of sysbench test
by Zhang Yu 09 Nov '20

09 Nov '20

[why] Use the special version of sysbench test mysql, use different test file are required [how] Add parameters to test, and use different test file of sysbench Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 79 +++++++++++++++++++++++++++++++------------- 1 file changed, 56 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..a4d79550 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,85 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=600}" +: "${report_interval:=1}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" + } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql + -- 2.23.0

1 0

Re: [PATCH compass-ci] container: fix failed to build kibana images
by Liu Yinsi 09 Nov '20

09 Nov '20

>> >> # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 >> >>diff --git a/container/kibana/build b/container/kibana/build >>index a7e4717..60fdea2 100755 >>--- a/container/kibana/build >>+++ b/container/kibana/build >>@@ -3,4 +3,10 @@ >> # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. >> # frozen_string_literal: true >> >>-system 'docker build -t kibana:7.6.2 .' >>+BASE_IMAGE_DICT = {'aarch64'=>'gagara/kibana-oss-arm64:7.6.2', >>+ 'x86_64'=>'kibana:7.6.2' > >add space surround '=>' > >cat the hash in multi lines, write it as: >BASE_IMAGE_DICT = { > 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', > 'x86_64' => 'kibana:7.6.2' >} good Thanks, Yinsi > >Thanks, >Luan Shengde > >>+} >>+ >>+BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] >>+ >>+system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." >>-- >>2.23.0 >>

1 0