- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sbin/compare: update help message
by Lu Weitao 22 Feb '21

22 Feb '21

[Why] the option -d/--dimensions no specific instructions [How] Usage: ... options: -d, --dimension dimensions this option activate group compare mode, require filter and dimenssion filter like: os=openeuler,centos suite=iperf dimensions like: "tbox_group os_arch os ..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/compare | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/sbin/compare b/sbin/compare index e8d1ac4..2ee3268 100755 --- a/sbin/compare +++ b/sbin/compare @@ -11,8 +11,8 @@ # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" # compare "os=debian" "os=centos" -c "suite=iperf" -# compare "os=centos" -d "os_version os_arch" -# compare "os=centos" -d "os_version os_arch" --theme="classic" +# compare os=centos -d "os_version os_arch" +# compare os=centos suite=iperf -d "os_version os_arch" --theme="classic" require 'optparse' require_relative '../lib/compare.rb' @@ -37,7 +37,8 @@ opt_parser = OptionParser.new do |opts| common_conditions = c end - opts.on('-d', '--dimension dimensions', 'dimensions to group compare: "tbox_group os_arch os ..."') do |d| + opts.on('-d', '--dimension dimensions', 'this option activate group compare mode, require filter and dimenssion', + 'filter like: os=openeuler,centos suite=iperf', 'dimensions like: "tbox_group os_arch os ..."') do |d| dimensions = d is_group = true end -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd-base/Dockerfile: fix failed to build container
by Liu Yinsi 22 Feb '21

22 Feb '21

[error] root@taishan200-2280-2s64p-256g--a7 /c/compass-ci/container/fluentd-base# ./build ... ERROR: Error installing async: ERROR: Failed to build gem native extension. current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r /usr/bin/ruby -I /usr/lib/ruby/2.6.0 -r ./siteconf20210222-482-mws95t.rb extconf.rb checking for unistd.h... yes checking for linux/aio_abi.h... no checking for linux/io_uring.h... no checking for sys/select.h... yes checking for port_event_t in poll.h... no checking for sys/epoll.h... yes checking for sys/event.h... no checking for port_event_t in port.h... no checking for sys/resource.h... yes creating Makefile current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r make "DESTDIR=" clean current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r make "DESTDIR=" compiling bytebuffer.c compiling monitor.c compiling nio4r_ext.c In file included from nio4r_ext.c:6: ../libev/ev.c:479:11: fatal error: linux/version.h: No such file or directory 479 | # include <linux/version.h> | ^~~~~~~~~~~~~~~~~ compilation terminated. make: *** [Makefile:245: nio4r_ext.o] Error 1 make failed, exit code 2 Gem files will remain installed in /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5 for inspection. Results logged to /usr/lib/ruby/gems/2.6.0/extensions/aarch64-linux/2.6.0/nio4r-2.5.5/gem_make.out The command '/bin/sh -c umask 002 && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && gem install fluentd && gem install fluent-plugin-rabbitmq && gem install fluent-plugin-elasticsearch && gem install fluent-plugin-tail-ex && gem install fluent-plugin-tail-multiline && gem install json && gem install async && gem install webrick && gem install io-console && gem install etc' returned a non-zero code: 1 [how] 'gem install async' in Dockerfile cause error, apk add linux-headers can fix it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/fluentd-base/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/fluentd-base/Dockerfile b/container/fluentd-base/Dockerfile index 08bcd9c..2d181f5 100644 --- a/container/fluentd-base/Dockerfile +++ b/container/fluentd-base/Dockerfile @@ -10,7 +10,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update RUN apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'linux-headers' RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: optimize os_mount.md
by Yu Chuan 22 Feb '21

22 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 207 +++++++++++++++++++++++++++++--------------- 1 file changed, 137 insertions(+), 70 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..ebe8566b9443 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,9 +1,9 @@ # Summary ========= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: +The below types are supported: - nfs - cifs - initramfs @@ -11,84 +11,151 @@ It has the following optional values: - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. +# os_mount=local +================ + +## description + +If you specifies os_mount=local, the rootfs type of your job will be a logical volume in the testbox. + +> logical volume format: `/dev/mapper/os-${os}_${os_arch}_${os_version}` + +> optional vars in ${kernel_append_root}: + - ${save_root_partition}, ${use_root_partition} + - purpose: + - when os_mount=local, rootfs is stored in the logical volume on the disk, + but it will be wiped by default in the preprocessing action performed by each job. + just is initrd stage. + - so we need a solution to solve scenarios that rootfs data needs to be persisted. + - how to implement: + - design a unique string for your job, it will be used as a unique symbol for your job. + the rootfs data of your job will be persistently saved to the corresponding testbox. + - so, when you need to use the rootfs data of this job in subsequent jobs, you only need + to use this unique string to submit subsequent-job.yaml, and then the subsequent-job.yaml + will be executed based on the prsistent rootfs data. + - how to use: + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + this job-20210218 so that it can be used by the subsequent jobs. So you need add the + follow field in your job-20210218.yaml. + ``` + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + ``` + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml, So you need add the follow field in your job-20210219.yaml + ``` + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + - description: + - ${save_root_partition} and ${use_root_partition} must be the same, and obviously, they need be unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox ``` dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} boot ``` 3. dracut step of boot ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi + rsync_src_lv() + { + lvdisplay ${SRC_LV} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${SRC_LV}) os || exit + + # rsync nfsroot to ${SRC_LV} + mount -t nfs ${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} /mnt + mkdir /mnt1 && mount ${SRC_LV} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${SRC_LV} to readonly + lvchange --permission r ${SRC_LV} + } + + check_src_lv() + { + lvdisplay ${SRC_LV} > /dev/null || exit + } + + snapshot_boot_lv() + { + [ "$SRC_LV" == "$BOOT_LV" ] && return + + lvremove --force ${BOOT_LV} + lvcreate --size 10G --name $(basename ${BOOT_LV}) --snapshot ${SRC_LV} || exit + } + + no_save_no_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03" + + rsync_src_lv ${SRC_LV} + } + + only_save() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + rsync_src_lv ${SRC_LV} + } + + only_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03" + + check_src_lv ${SRC_LV} + } + + save_and_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + check_src_lv ${SRC_LV} + } + + main() + { + [ -z ${use_root_partition} ] && [ -z ${save_root_partition} ] && no_save_no_use + + [ -z ${use_root_partition} ] && [ -n ${save_root_partition} ] && only_save + + [ -n ${use_root_partition} ] && [ -z ${save_root_partition} ] && only_use + + [ -n ${use_root_partition} ] && [ -n ${save_root_partition} ] && save_and_use + + snapshot_boot_lv + boot_from_lv + } + + main() ``` 4. execute the job -- 2.23.0

2 4

[PATCH compass-ci] doc: update document
by Luan Shengde 22 Feb '21

22 Feb '21

rename key word: my oss commit => my_oss_commit Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 4 ++-- doc/manual/apply-account.md | 6 +++--- doc/manual/apply-account.zh.md | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 6158b65..4d38e78 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -17,9 +17,9 @@ send mail to: compass-ci-robot(a)qq.com subject: apply account mail content - my oss commit: commit_url + my_oss_commit: commit_url example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + my_oss_commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… attahcment ssh pub_key diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 974fe63..542251d 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -4,7 +4,7 @@ You'll need an account to submit jobs. The account is mainly for tracking your r We have limited pool of testboxes, so there have to be some usage control. We offer free accounts to OSS (Open Source Software) contributors where the OSS project is in our test coverage. -You can prove yourself as OSS contributor by providing "my oss commit" in the below email. +You can prove yourself as OSS contributor by providing "my_oss_commit" in the below email. We offer free accounts to our collaborators too. If you are one of our collaborators, you should know the collaboration channel to apply account. @@ -19,7 +19,7 @@ Email template: To: compass-ci-robot(a)qq.com Subject: apply account - my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + my_oss_commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} {{ attach your ssh pubkey file(s), e.g. ~/.ssh/id_rsa.pub }} @@ -29,7 +29,7 @@ Email template: - the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. -- my oss commit: a git commit URL that has your name and email. +- my_oss_commit: a git commit URL that has your name and email. The git repo should be in our test coverage and registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). If your contributed OSS project is not in upstream-repos, you may refer to diff --git a/doc/manual/apply-account.zh.md b/doc/manual/apply-account.zh.md index 5e6605d..849364f 100644 --- a/doc/manual/apply-account.zh.md +++ b/doc/manual/apply-account.zh.md @@ -4,7 +4,7 @@ 因为目前我们的测试机资源池有限，因此需要做一些使用限制。我们为平台测试所覆盖的开源软件（OSS）的贡献者提供了免费的账号。 -你可以通过在下面邮件中提供‘my oss commit’来验明正身您是一个开源软件（OSS）的贡献者。 +你可以通过在下面邮件中提供‘my_oss_commit’来验明正身您是一个开源软件（OSS）的贡献者。我们同时也为我们的合作者提供了免费的账号。这种情况需要提供用的邮件，名字，ssh公钥，以及申请测试机的目的。 @@ -18,7 +18,7 @@ 收件人: compass-ci-robot(a)qq.com 邮件标题: apply account - my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + my_oss_commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} {{ 将您的ssh公钥添加为邮件附件，例如 ~/.ssh/id_rsa.pub }} @@ -26,7 +26,7 @@ - 使用您的信息替换{{ }}中的内容 - 邮件名应该使用英文或中文拼音，例如模板中的“David Rientjes” -- my oss commit: 使用您的邮件和名字提交的代码的地址。 +- my_oss_commit: 使用您的邮件和名字提交的代码的地址。您的仓库需要在平台测试覆盖的范围内，并且注册到： -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends/sysbench-mysql
by Wei Jihui 22 Feb '21

22 Feb '21

1. when pack pkg/sysbench-mysql, it need mysql so add libmariadb-dev(mariadb-devel) at sysbench-mysql-dev 2. when exec test/sysbench-mysql, it will install mysql at client, no need install as depends, so delete libmariadb3 at sysbench-mysql Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/sysbench-mysql | 1 - distro/depends/sysbench-mysql-dev | 3 ++- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index b0d33ffe..a248fd07 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,4 +1,3 @@ libnuma1 xfsprogs ethtool -libmariadb3 diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev index 65633b33..992db26d 100644 --- a/distro/depends/sysbench-mysql-dev +++ b/distro/depends/sysbench-mysql-dev @@ -1,2 +1,3 @@ +libtool pkg-config -mysql-server +libmariadb-dev -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix fs parameter parse
by Wei Jihui 22 Feb '21

22 Feb '21

[input] fs: ext4 [error] this fs will parse error by lkp-tests/params/fs fs: fs: &1 fs: ext4 [why] job.rb will parse fs twice by lkp-test/params/fs: /(.*)/{ fs: $1} this will create two fs. [how] only parse all parameters once, that is only exec expand_params once. not at the first time is for job should split array params at the first time. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec9..7fc7cb5b 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -447,7 +447,7 @@ class Job job = deepcopy(@job) job['___'] = nil @jobx = job - if @job.include?(:expand_params) + if @job.include?(:expand_params) && !first_time expand_params @job = deepcopy(@jobx) @job.delete('___') -- 2.23.0

2 1

[PATCH v2 compass-ci 1/4] container/mail-robot: add handle forwarded my_name/my_email(s)
by Luan Shengde 22 Feb '21

22 Feb '21

[why] for forwarded email from account issuer, the email content will contain my_name/my_email(s). a forwarded email may contain multi my_name/my_email(s), just loop them. the forwarded email content may like: --- my_name: name_1 my_email: email_1 account_vm: true/false/yes/no --- my_name: name_2 my_email: email_2 --- the replied 'apply account' email will return login command for account VM server if account_vm has a value: yes/true add store lab to ES Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 65 ++++++++++++++++++++--- 1 file changed, 57 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index e8288d7..818f900 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -55,6 +55,9 @@ class ApplyAccount @es_host = @send_mail_host @es_port = ES_PORT + # email info file for account issuers. + @account_issuer = YAML.load_file '/etc/compass-ci/account-issuer.yaml' + @my_info = {} end @@ -63,11 +66,35 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] - parse_mail_content - apply_my_account - store_account_info - send_mail('') + # for the forwarded email, it contains my_email/my_name .etc. + # all emails from account issuer will be treated as forwarded emails. + # the mail_content may like: + # --- + # my_name: name_1 + # my_name: email_1 + # account_vm: true/false/yes/no + # --- + # my_name: name_2 + # my_email: email_2 + # --- + # the forwarded email allowed to contain multi my_email/my_name(s) + # in this case, we will loop them + if @account_issuer.include? @my_info['my_email'] + users_info = forward_users + users_info.each do |user_info| + # for forwarded email for multi user, avoid rezidual information from the last. + @my_info.clear + assign_account_vm = user_info.delete('account_vm') + @my_info.update user_info + + applying_account(assign_account_vm) + sleep 5 + end + else + parse_mail_content + applying_account(false) + end rescue StandardError => e puts e.message puts e.backtrace @@ -75,6 +102,24 @@ class ApplyAccount send_mail(e.message) end + def forward_users + forward_email_content = ParseApplyAccountEmail.new(@mail_content) + users_info = forward_email_content.extract_users + + users_info.clone.each_index do |i| + users_info[i]['my_ssh_pubkey'] = [] + users_info[i]['lab'] = ENV['lab'] + end + + users_info + end + + def applying_account(assign_account_vm) + account_info = apply_my_account + store_account_info + send_mail('', account_info, assign_account_vm) + end + def parse_mail_content parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) @@ -93,7 +138,9 @@ class ApplyAccount apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + if my_ssh_pubkey_new + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + end @my_info.update apply_info apply_info['is_update_account'] = true apply_info @@ -102,6 +149,8 @@ class ApplyAccount def apply_my_account my_account_es = read_my_account_es apply_info = {} + apply_info['enable_login'] = true + if my_account_es build_apply_info(apply_info, my_account_es) else @@ -109,7 +158,6 @@ class ApplyAccount @my_info['my_token'] = my_token apply_info.update @my_info end - apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -118,6 +166,7 @@ class ApplyAccount acct_info = apply_account.apply_jumper_account @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + acct_info end def store_account_info @@ -125,9 +174,9 @@ class ApplyAccount es.put_source_by_id(@my_info['my_email'], @my_info) end - def send_mail(error_message) + def send_mail(error_message, account_info, assign_account_vm) email_message = if error_message.empty? - build_apply_account_email(@my_info) + build_apply_account_email(@my_info, account_info, assign_account_vm) else build_apply_account_fail_email(@my_info, error_message) end -- 2.23.0

2 4

[PATCH compass-ci] doc: optimize os_mount.md
by Yu Chuan 21 Feb '21

21 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 181 +++++++++++++++++++++++++++++--------------- 1 file changed, 120 insertions(+), 61 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..b06b27237c75 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,9 +1,9 @@ # Summary ========= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: +The below types are supported: - nfs - cifs - initramfs @@ -11,84 +11,143 @@ It has the following optional values: - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local + - submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + - submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + - submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + - submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + - submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local -# Optional Values -================= +# os_mount=local +================ -## Work flow when os_mount=local +## description -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. +If you specifies os_mount=local, the rootfs type of your job will be a logical volume in the testbox. - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. +> logical volume format: `/dev/mapper/os-${os}_${os_arch}_${os_version}` + +> optional vars in ${kernel_append_root}: + - ${save_root_partition}, ${use_root_partition} + - you can save the rootfs data of any job, and use it in subsequent job. + - ${save_root_partition} and ${use_root_partition} must be the same, and obviously, they need be unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + - demo kernel_append_root field of job.yaml: + ``` + # demo 1. save the rootfs data of this job + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210219 + + # demo 2. use the rootfs data of previous job + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + + # demo 3. use the rootfs data of previous job, and save the rootfs data of this job + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` + +## work flow + +1. user submit job + - job fields: + - os: openeuler + - os_arch: aarch64 + - os_version: 20.03 + - os_mount: local + - kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 2. scheduler return the ipxe_str to testbox ``` dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} boot ``` 3. dracut step of boot ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { + copy_nfsroot_to_src_lv() + { + src_lv=$1 + + lvdisplay ${src_lv} > /dev/null || { # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 + lvcreate -L 10G -n $(basename ${src_lv}) os || exit 1 - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 + # rsync nfsroot to ${src_lv} + mount -t nfs ${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 cp -a /mnt/. /mnt1/ umount /mnt /mnt1 - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} + # change premission of ${src_lv} to readonly + lvchange -p r ${src_lv} } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi + } + + check_src_lv() + { + lvdisplay $1 > /dev/null || exit 1 + } + + clone_boot_lv_from_src_lv() + { + src_lv=$1 + boot_lv=$2 + + lvremove -f ${boot_lv} + lvcreate ${boot_lv} from ${src_lv} || exit 1 + } + + no_save_no_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + + copy_nfsroot_to_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + only_save() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + copy_nfsroot_to_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + only_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + + check_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + save_and_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + check_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + main() + { + [ -z ${use_root_partition} ] && [ -z ${save_root_partition} ] && no_save_no_use + + [ -z ${use_root_partition} ] && [ -n ${save_root_partition} ] && only_save + + [ -n ${use_root_partition} ] && [ -z ${save_root_partition} ] && only_use + + [ -n ${use_root_partition} ] && [ -n ${save_root_partition} ] && save_and_use + } + + main() ``` 4. execute the job -- 2.23.0

2 10

[PATCH v2 compass-ci] sbin/build-my-info: fix no key: my_ssh_pubkey
by Luan Shengde 20 Feb '21

20 Feb '21

[why] it will read my_ssh_pubkey and check it when submit jobs. $ submit -m -c job.yaml -i ~/lkp-tests/jobs/ssh-on-fail.yaml submit_id=f87d2951-c226-4e51-a3e4-c71818a4c974 submit job.yaml failed, got job id=0, error: Missing hash key: "my_ssh_pubkey" [how] add initialize my_ssh_pubkey when building my_info. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index c7a7798..ecfcb42 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -12,6 +12,7 @@ def build_my_info(option) info_es = build_my_info.search_my_info info_es.update option info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? + info_es['my_ssh_pubkey'] = [] build_my_info.config_my_info(info_es) end -- 2.23.0

1 0

[PATCH v2 compass-ci] answerback-email: add store lab to ES
by Luan Shengde 20 Feb '21

20 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 8dceee1..f573f0f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -40,7 +40,8 @@ my_info = { 'my_commit_url' => nil, 'my_token' => nil, 'my_login_name' => nil, - 'my_ssh_pubkey' => [] + 'my_ssh_pubkey' => [], + 'lab' => LAB } # stdin_info is used to store infos added with option: @@ -179,7 +180,6 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info - apply_info['lab'] = LAB assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" account_info_str = RestClient.post assign_account_url, apply_info.to_json -- 2.23.0

1 0

[PATCH compass-ci] auto-submit: use the lib script to config the account
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/auto_submit | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 6d29543..b408158 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,8 +7,7 @@ require 'bunny' require 'json' require 'yaml' require 'set' -require_relative '../lib/build_my_info_client' -require "#{ENV['CCI_SRC']}/container/defconfig" +require_relative '../lib/config_account' # receive message and auto submit job class AutoSubmit @@ -130,23 +129,7 @@ class AutoSubmit end end -def my_email - names = Set.new %w[auto-submit] - - defaults = relevant_defaults(names) - defaults['auto-submit']['my_email'] -end - -def config_yaml - build_my_info = BuildMyInfo.new(my_email) - - my_info = build_my_info.search_my_info - build_my_info.config_default_yaml(my_info) - build_my_info.config_lab_yaml(my_info) -end - -# fix verification problems when submit jobs by auto-submit service -config_yaml +config_yaml('auto-submit') auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] ner/mail-robot: optimize start
by Luan Shengde 20 Feb '21

20 Feb '21

attach the directory instead of the single file. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/start b/container/mail-robot/start index 3827e2c..bd2a62d 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -31,7 +31,7 @@ cmd = %w[ ] + env + %W[ -v /c/upstream-repos:/c/upstream-repos:ro -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw - -v /etc/compass-ci/conf/sec_admins.yaml:/etc/compass-ci/conf/sec_admins.yaml:ro + -v /etc/compass-ci:/etc/compass-ci:ro -v #{ENV['CCI_SRC']}:/c/compass-ci:rw -w /c/compass-ci/container/mail-robot mail-robot -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/mail-robot: add login command
by Luan Shengde 20 Feb '21

20 Feb '21

add login command for account VM server. case user info has key account_vm with value: yes/true, the email will send the login command. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b71b3fe..8e3aa6e 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -3,13 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_email(my_info) +def build_apply_account_email(my_info, account_info, assign_account_vm) email_greeting = if my_info['bisect'] 'We have automatically created the following information for you.' else 'Thank you for joining us.' end + logging_msg = <<~LOGIN_MSG + + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + LOGIN_MSG + + logging_account_vm = assign_account_vm ? logging_msg : '' + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready @@ -17,7 +30,7 @@ def build_apply_account_email(my_info) Dear #{my_info['my_name']}, #{email_greeting} - + #{logging_account_vm} You can use the following info to submit jobs: notice: -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/mail-robot: add parsing forwarded email
by Luan Shengde 20 Feb '21

20 Feb '21

for forwarded 'apply account' email, my_email/my_name is store in a fixed format in the mail content. extract a list of users info from the mail content Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/parse-apply-account-email.rb | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 92db1da..3de2fb5 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -57,9 +57,18 @@ class ParseApplyAccountEmail end def extract_users - users = extract_mail_content_body.split(/\r|\n/) + users_info = [] + + users = extract_mail_content_body.split(/---+/) users.delete('') - users + users.each do |user| + user_info = YAML.safe_load(user) + next unless user_info.include?('my_email') + + users_info << user_info + end + + return users_info end def extract_commit_url -- 2.23.0

1 0

[PATCH compass-ci] delimter: Use delimiter account submit the bisect job
by Cao Xueliang 20 Feb '21

20 Feb '21

Add the config_account.rb to create the account info in container, it will use the specified account when submit the job in container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 1 + lib/config_account.rb | 22 ++++++++++++++++++++++ src/delimiter.rb | 2 ++ 3 files changed, 25 insertions(+) create mode 100644 lib/config_account.rb diff --git a/container/delimiter/start b/container/delimiter/start index 669923b..3cff23a 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -31,6 +31,7 @@ cmd = %w[ -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v /etc/localtime:/etc/localtime:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -w #{DEFAULT_CCI}/src delimiter ] diff --git a/lib/config_account.rb b/lib/config_account.rb new file mode 100644 index 0000000..dc0067c --- /dev/null +++ b/lib/config_account.rb @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "./build_my_info_client" +require_relative "../container/defconfig" + +def my_email(account) + names = %W(#{account}) + + defaults = relevant_defaults(names) + defaults[account]['my_email'] +end + +def config_yaml(account) + build_my_info = BuildMyInfo.new(my_email(account)) + + my_info = build_my_info.search_my_info + build_my_info.config_default_yaml(my_info) + build_my_info.config_lab_yaml(my_info) +end diff --git a/src/delimiter.rb b/src/delimiter.rb index bd40775..1ea5de5 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -3,8 +3,10 @@ # frozen_string_literal: true require_relative './delimiter/delimiter' +require_relative '../lib/config_account' begin + config_yaml('delimiter') delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: fix undefined method `/'
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index d43785b..104f098 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -47,7 +47,7 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, We found some error/warning(s) and the first bad commit in the below project: - git url: #{@upstream_url/@commit_id} + git url: #{@upstream_url}/#{@commit_id} git commit: #{@commit_id} ("#{(a)git_commit.subject}") All error/warning(s) (new ones prefixed by >>): -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sparrow: separate service config
by Cao Xueliang 20 Feb '21

20 Feb '21

Extract service config from defaults/yaml to the service/yaml. Read service/yaml in defconfig.rb and defconfig.sh. Add scheduler/yaml for scheduler set the content to the job. Adjust the order of service config. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 1 + sparrow/3-code/dev-env | 80 +++++++++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 25 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 602b498..b121de8 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/service/*.yaml', '/etc/compass-ci/accounts/*.yaml', '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| diff --git a/container/defconfig.sh b/container/defconfig.sh index 7c575ef..c33a790 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -10,6 +10,7 @@ load_cci_defaults() yaml_file=( /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/service/*.yaml /etc/compass-ci/accounts/*.yaml "$HOME"/.config/compass-ci/defaults/*.yaml ) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8812839..cc94e01 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -11,52 +11,82 @@ server_ip=$(awk '/^server_ip:\s/ {print $2; exit}' /etc/compass-ci/setup.yaml) server_name=$(hostname | cut -f1 -d.) : ${sched_host:=$server_ip} : ${sched_port:=3000} -: ${os_http_host:=$server_ip} -: ${initrd_http_host:=$server_ip} -: ${initrd_http_port:=8800} +: ${ES_HOST:=$server_ip} +: ${ES_PORT:=9200} +: ${OS_HTTP_HOST:=$server_ip} +: ${SEND_MAIL_HOST:=$server_ip} +: ${SEND_MAIL_PORT:=49000} +: ${SRV_HTTP_HOST:=$server_ip} +: ${SRV_HTTP_PORT:=11300} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} +: ${INITRD_HTTP_HOST:=$server_ip} +: ${INITRD_HTTP_PORT:=8800} +: ${ASSIST_RESULT_HOST:=$server_ip} +: ${ASSIST_RESULT_PORT:=8102} +: ${RESULT_WEBDAV_HOST:=$server_ip} +: ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} -: ${LOGGING_ES_HOST:=$server_ip} -: ${LOGGING_ES_PORT:=9202} -: ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} -: ${ASSIST_RESULT_PORT:=8102} -: ${RESULT_WEBDAV_PORT:=3080} -: ${ES_HOST:=$server_ip} -: ${ES_PORT:=9200} -: ${SRV_HTTP_PORT:=11300} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +lab: nolab +EOF + +mkdir -p /etc/compass-ci/service +cat > /etc/compass-ci/service/service-env.yaml <<EOF +ES_HOST: $ES_HOST +ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port LKP_SERVER: $server_ip GIT_SERVER: $GIT_SERVER -TASKQUEUE_HOST: $sched_host -OS_HTTP_HOST: $os_http_host -INITRD_HTTP_HOST: $initrd_http_host -INITRD_HTTP_PORT: $initrd_http_port +OS_HTTP_HOST: $OS_HTTP_HOST GITCACHE_HOST: $gitcache_host GITCACHE_PORT: $gitcache_port +SRV_HTTP_HOST: $SRV_HTTP_HOST +SRV_HTTP_PORT: $SRV_HTTP_PORT +TASKQUEUE_HOST: $sched_host +SEND_MAIL_HOST: $SEND_MAIL_HOST +SEND_MAIL_PORT: $SEND_MAIL_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT +INITRD_HTTP_HOST: $INITRD_HTTP_HOST +INITRD_HTTP_PORT: $INITRD_HTTP_PORT +ASSIST_RESULT_HOST: $ASSIST_RESULT_HOST +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_HOST: $RESULT_WEBDAV_HOST +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT -LOGGING_ES_HOST: $LOGGING_ES_HOST -LOGGING_ES_PORT: $LOGGING_ES_PORT -SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT -LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci -ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT -RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT -lab: nolab -ES_HOST: $ES_HOST -ES_PORT: $ES_PORT -SRV_HTTP_PORT: $SRV_HTTP_PORT +LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +EOF + +mkdir -p /etc/compass-ci/scheduler +cat > /etc/compass-ci/scheduler/local-testbox-env.yaml <<-EOF +LKP_SERVER: +GIT_SERVER: +GITCACHE_HOST: +GITCACHE_PORT: +SRV_HTTP_HOST: +SRV_HTTP_PORT: +SEND_MAIL_HOST: +SEND_MAIL_PORT: +INITRD_HTTP_HOST: +INITRD_HTTP_PORT: +ASSIST_RESULT_HOST: +ASSIST_RESULT_PORT: +RESULT_WEBDAV_HOST: +RESULT_WEBDAV_PORT: EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler: add testbox env in scheduler
by Cao Xueliang 20 Feb '21

20 Feb '21

We add the testbox env use the submit script in lkp-tests, but the submit don't need to do that, so, we add the testbox env in scheduler. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 ++ src/scheduler/testbox_env.cr | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 src/scheduler/testbox_env.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index 1afc22f..c3eeaf4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "scheduler/testbox_env.cr" require "../scheduler/elasticsearch_client" require "./json_logger" @@ -52,6 +53,7 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new diff --git a/src/scheduler/testbox_env.cr b/src/scheduler/testbox_env.cr new file mode 100644 index 0000000..bf8cd8a --- /dev/null +++ b/src/scheduler/testbox_env.cr @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + def get_service_env + hash = Hash(String, JSON::Any).new + yaml_any = File.open("/etc/compass-ci/service/service-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + return Hash(String, JSON::Any).from_json(yaml_any.to_json) + end + + def testbox_env(flag = "local") + service_env = get_service_env + hash = Hash(String, JSON::Any).new + + yaml_any = File.open("/etc/compass-ci/scheduler/#{flag}-testbox-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + hash.merge!(Hash(String, JSON::Any).from_json(yaml_any.to_json)) + hash.each do |key, value| + if value == nil + hash[key] = service_env[key] + end + end + + hash + end +end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: modify the function get_hosts_file
by Li Ping 20 Feb '21

20 Feb '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [reason]: get_hosts_file use tbox_group to search for hosts_file --- lib/job.rb | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec95..8f8c91b28 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,17 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_host = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts" + lkp_host = "#{LKP_SRC}/hosts" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + check_files = ["#{lab_host}/#{@job['testbox']}", + "#{lkp_host}/#{@job['testbox']}", + "#{lab_host}/#{tbox_group}", + "#{lkp_host}/#{tbox_group}"] + + check_files.each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

2 2

[PATCH lkp-tests] tests/rpmbuild-pkg: various cleanup
by Wu Fengguang 20 Feb '21

20 Feb '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- tests/rpmbuild-pkg | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 781d04a0d..9f455f0d0 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,8 +5,8 @@ . $LKP_SRC/lib/upload.sh [ -n "$rpm_repo" ] || die "rpm_repo is empty" -benchmark=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark}" +package_name=${rpm_repo##*/} +sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() { @@ -16,15 +16,15 @@ init_workspace() get_pkgfile() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" } download_rpm_repo() { - filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in ${filelist[*]} + for pkgfile in "${filelist[@]}" do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" @@ -34,20 +34,29 @@ download_rpm_repo() build_rpm() { - spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_dir` + local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec + + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + # Install build depends - yum-builddep -y $spec_dir + yum-builddep -y $spec_file || exit # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_dir + spectool -g -R $spec_file || exit # Building rpm or srpm packages - rpmbuild -ba $spec_dir + rpmbuild -ba $spec_file || exit +} + +show_rpm_files() +{ + find ${HOME}/rpmbuild/RPMS -type f -name "*.rpm" + find ${HOME}/rpmbuild/SRPMS -type f -name "*.rpm" } upload_rpm_pkg() { - # rpm package will be generated in "${HOME}/rpmbuild/SRPMS" and "${HOME}/rpmbuild/RPMS" - for file in $(find ${HOME}/rpmbuild/ -type f -name "*.rpm") + local file + for file in $(show_rpm_files) do upload_one_curl ${file} ${sync_dest} done -- 2.23.0

1 0

[PATCH v2 compass-ci] rsync-server: add support upload function for ks file
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] We need to achieve automatic installation through 'pxe + kickstart + dnsmasq + httpd'. So we need to rsync the ks file by rsync-server. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/rsync-server/rsyncd.conf | 6 ++++++ container/rsync-server/start | 1 + 2 files changed, 7 insertions(+) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index cdba89cf20c8..5a9bd2ddadec 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -10,3 +10,9 @@ path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete + +[ks] +path = /srv/os/install/ks/ +read only = no +max connections = 200 +refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 956013268f89..334f5c2c4a69 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -11,6 +11,7 @@ cmd=( --restart=always --name=rsync_server -v /srv/result/upload:/srv/result/upload + -v /srv/os/install/ks:/srv/os/install/ks -v /etc/localtime:/etc/localtime:ro -p 11387:11387 rsync_server:latest -- 2.23.0

1 0

[PATCH lkp-tests] bootstrap.sh: add a test ping condition to network_ok
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] In physical machine, there will be a delay between the start of network card and the availability of the network. So we add a test ping condition to network_ok. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 06aa09bf65b2..38965ff60755 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -85,14 +85,19 @@ net_devices_link() done } +test_ping() +{ + ping -c 1 -W 1 $LKP_SERVER > /dev/null || return 1 +} + network_ok() { local i for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && test_ping && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && test_ping && return 0 done is_clearlinux && { -- 2.23.0

2 7

[PATCH compass-ci] lib/git_mirror.rb: change the way to load DEFAULTS file
by Li Yuanchao 19 Feb '21

19 Feb '21

Before: DEFAULTS files are individual. After: Load DEFAULTS file and merge it with the parent directory DEFAULTS files. So the outer DEFAULTS files would keep common things for the inner ones. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 39b1619..346352b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -130,6 +130,7 @@ class MirrorMain defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) end def traverse_repodir(repodir) @@ -224,7 +225,7 @@ class MirrorMain git_repo = "#{project}/#{fork_name}" @git_info[git_repo] = YAML.safe_load(File.open(repodir)) @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo) + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -449,15 +450,17 @@ end # main thread class MirrorMain - def merge_defaults(git_repo) - defaults_key = File.dirname(git_repo) + def merge_defaults(object_key, object) + return object if object_key == 'default' + + defaults_key = File.dirname(object_key) while defaults_key != '.' - return @defaults[defaults_key].merge(@git_info[git_repo]) if @defaults[defaults_key] + return @defaults[defaults_key].merge(object) if @defaults[defaults_key] defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(@git_info[git_repo]) if @defaults['default'] + return @defaults['default'].merge(object) if @defaults['default'] - return @git_info[git_repo] + return object end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] lkp-bootstrap: delete unnecessary exit
by Xu Xijian 19 Feb '21

19 Feb '21

The exit cmd after reboot is unnecessary, delete it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- rootfs/addon/etc/init.d/lkp-bootstrap | 1 - 1 file changed, 1 deletion(-) diff --git a/rootfs/addon/etc/init.d/lkp-bootstrap b/rootfs/addon/etc/init.d/lkp-bootstrap index f66ec4eb5..091b29890 100755 --- a/rootfs/addon/etc/init.d/lkp-bootstrap +++ b/rootfs/addon/etc/init.d/lkp-bootstrap @@ -24,7 +24,6 @@ if [ -s /lkp/run/lkp-bootstrap.pid ]; then # the previous lkp-bootstrap init.d script failed to restart the machine echo "LKP bootstrap: first run may failed, retry rebooting" reboot - exit fi read_kernel_cmdline_vars() -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] bin/lkp-setup-rootfs: refactor the chmod_root_ssh()
by Xu Xijian 19 Feb '21

19 Feb '21

Add blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cf48c8f6b..bc0f04b2a 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -245,6 +245,7 @@ chmod_root_ssh() { [ -d "/root/.ssh" ] || return [ -f "/root/.ssh/id_rsa" ] || return + chmod 700 /root/.ssh/ chmod 600 /root/.ssh/id_rsa } -- 2.23.0

2 2