January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 27 Jan '21

27 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..2532988 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,25 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 2

[PATCH lkp-tests] stats: add output failed msg
by Zhang Yale 27 Jan '21

27 Jan '21

[Why] Output meed to be displayed in the ansible_test.json file. [Input] fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined\n\nThe error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Define mysql_packages.\n ^ here\n"} PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 [Output] playbook.run.msg: The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined The error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Define mysql_packages. ^ here nr_ok: 2 nr_changed: 0 nr_failed: 1 nr_skipped: 1 nr_rescued: 0 nr_ignored: 0 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3..f0766412 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,17 @@ #!/usr/bin/env ruby +require 'json' while (line = STDIN.gets) case line.chomp! + when /FAILED!\s+=>\s+(\{.*\})/ + value = JSON.load($1)['msg'] + puts "playbook.run.msg: #{value}" when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + puts "nr_ok: #{$1}" + puts "nr_changed: #{$2}" + puts "nr_failed: #{$4}" + puts "nr_skipped: #{$5}" + puts "nr_rescued: #{$6}" + puts "nr_ignored: #{$7}" end end -- 2.23.0

2 2

[PATCH compass-ci] sbin/build-my-info: optimize options
by Luan Shengde 27 Jan '21

27 Jan '21

add output usage add option -h|--help set default option to -h|--help Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..028fbac 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -14,6 +14,11 @@ option = { } options = OptionParser.new do |opts| + opts.banner = "Usage: ./build-my-info.rb -e email -n name -l lab\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| option[:my_email] = email end @@ -25,8 +30,15 @@ options = OptionParser.new do |opts| opts.on('-l lab', 'lab') do |lab| option[:lab] = lab end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end end +ARGV << '-h' if ARGV.empty? + options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -- 2.23.0

1 0

[PATCH v1 compass-ci 2/2] providers/libvirt: split common elements of domain.xml
by Xiao Shenwei 27 Jan '21

27 Jan '21

[why] some elements can be specifies by user. case 1: a complete domain.xml we should replace some elements of user's xml like os name ... case 2: default domain.xml generate a default domain.xml case 3: only specifies a cpu templates use user's cpu template replace default cpu template case 4: composition template user can specifies different tempaltes like cpu, disk, ... [how] split domain.xml then template combination based on user requirements Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/templates/active.xml | 3 +++ providers/libvirt/templates/clock.xml | 6 ++++++ providers/libvirt/templates/cpu.xml | 6 ++++++ providers/libvirt/templates/devices.xml | 3 +++ providers/libvirt/templates/disk.xml | 7 +++++++ providers/libvirt/templates/domain.xml | 10 ++++++++++ providers/libvirt/templates/emulator.xml | 1 + providers/libvirt/templates/interface.xml | 6 ++++++ providers/libvirt/templates/memory.xml | 1 + providers/libvirt/templates/name.xml | 1 + providers/libvirt/templates/os.xml | 8 ++++++++ providers/libvirt/templates/seclabel.xml | 4 ++++ providers/libvirt/templates/serial.xml | 4 ++++ 13 files changed, 60 insertions(+) create mode 100644 providers/libvirt/templates/active.xml create mode 100644 providers/libvirt/templates/clock.xml create mode 100644 providers/libvirt/templates/cpu.xml create mode 100644 providers/libvirt/templates/devices.xml create mode 100644 providers/libvirt/templates/disk.xml create mode 100644 providers/libvirt/templates/domain.xml create mode 100644 providers/libvirt/templates/emulator.xml create mode 100644 providers/libvirt/templates/interface.xml create mode 100644 providers/libvirt/templates/memory.xml create mode 100644 providers/libvirt/templates/name.xml create mode 100644 providers/libvirt/templates/os.xml create mode 100644 providers/libvirt/templates/seclabel.xml create mode 100644 providers/libvirt/templates/serial.xml diff --git a/providers/libvirt/templates/active.xml b/providers/libvirt/templates/active.xml new file mode 100644 index 0000000..8ed7ee4 --- /dev/null +++ b/providers/libvirt/templates/active.xml @@ -0,0 +1,3 @@ +<on_poweroff>destroy</on_poweroff> +<on_reboot>destroy</on_reboot> +<on_crash>destroy</on_crash> diff --git a/providers/libvirt/templates/clock.xml b/providers/libvirt/templates/clock.xml new file mode 100644 index 0000000..7646c91 --- /dev/null +++ b/providers/libvirt/templates/clock.xml @@ -0,0 +1,6 @@ +<clock offset='utc'> + <timer name='rtc' tickpolicy='catchup' track='guest'/> + <timer name='hpet' present='no'/> + <timer name='pit' tickpolicy='delay'/> +</clock> + diff --git a/providers/libvirt/templates/cpu.xml b/providers/libvirt/templates/cpu.xml new file mode 100644 index 0000000..6991457 --- /dev/null +++ b/providers/libvirt/templates/cpu.xml @@ -0,0 +1,6 @@ +<vcpu placement='static'><%= nr_cpu %></vcpu> +<cpu mode='host-passthrough' check='none'> +  +</cpu> diff --git a/providers/libvirt/templates/devices.xml b/providers/libvirt/templates/devices.xml new file mode 100644 index 0000000..83cd847 --- /dev/null +++ b/providers/libvirt/templates/devices.xml @@ -0,0 +1,3 @@ +<devices> + <graphics type='vnc' autoport='yes' listen='0.0.0.0'><graphics> +</devices> diff --git a/providers/libvirt/templates/disk.xml b/providers/libvirt/templates/disk.xml new file mode 100644 index 0000000..7aecef0 --- /dev/null +++ b/providers/libvirt/templates/disk.xml @@ -0,0 +1,7 @@ +<% if disk %> +<disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='<%= disk %>'/> + <target dev='vda' bus='virtio'/> +</disk> +<% end %> diff --git a/providers/libvirt/templates/domain.xml b/providers/libvirt/templates/domain.xml new file mode 100644 index 0000000..a3fdb63 --- /dev/null +++ b/providers/libvirt/templates/domain.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" character="utf-8"?> +<domain type='kvm'> + <resource> + <partition>/machine</partition> + </resource> + <features> + <acpi/> + <gic version='3'/> + </features> +</domain> diff --git a/providers/libvirt/templates/emulator.xml b/providers/libvirt/templates/emulator.xml new file mode 100644 index 0000000..e915215 --- /dev/null +++ b/providers/libvirt/templates/emulator.xml @@ -0,0 +1 @@ +<emulator><%=qemu_path%></emulator> diff --git a/providers/libvirt/templates/interface.xml b/providers/libvirt/templates/interface.xml new file mode 100644 index 0000000..5d4c04f --- /dev/null +++ b/providers/libvirt/templates/interface.xml @@ -0,0 +1,6 @@ +<interface type='bridge'> + <source bridge='br0'/> + <mac address='<%= mac %>'/> + <target dev='vnet0'/> + <model type='virtio'/> +</interface> diff --git a/providers/libvirt/templates/memory.xml b/providers/libvirt/templates/memory.xml new file mode 100644 index 0000000..34aec49 --- /dev/null +++ b/providers/libvirt/templates/memory.xml @@ -0,0 +1 @@ +<memory unit='GiB'><%= memory.to_i %></memory> diff --git a/providers/libvirt/templates/name.xml b/providers/libvirt/templates/name.xml new file mode 100644 index 0000000..5dc98f4 --- /dev/null +++ b/providers/libvirt/templates/name.xml @@ -0,0 +1 @@ +<name><%= job_id %></name> diff --git a/providers/libvirt/templates/os.xml b/providers/libvirt/templates/os.xml new file mode 100644 index 0000000..317866d --- /dev/null +++ b/providers/libvirt/templates/os.xml @@ -0,0 +1,8 @@ +<os> + <type arch='<%= arch %>' machine='virt'>hvm</type> + <loader readonly='yes' type='pflash'><%= loader %></loader> + <kernel><%= kernel %></kernel> + <initrd><%= initrd %></initrd> + <cmdline><%= cmdline %></cmdline> + <nvram>/var/lib/libvirt/qemu/nvram/<%= job_id %>.fd</nvram> +</os> diff --git a/providers/libvirt/templates/seclabel.xml b/providers/libvirt/templates/seclabel.xml new file mode 100644 index 0000000..5c10a77 --- /dev/null +++ b/providers/libvirt/templates/seclabel.xml @@ -0,0 +1,4 @@ +<seclabel type='dynamic' model='dac' relabel='yes'> + <label>+0:+0</label> + <imagelabel>+0:+0</imagelabel> +</seclabel> diff --git a/providers/libvirt/templates/serial.xml b/providers/libvirt/templates/serial.xml new file mode 100644 index 0000000..2edc882 --- /dev/null +++ b/providers/libvirt/templates/serial.xml @@ -0,0 +1,4 @@ +<serial type='file'> + <source path="<%= log_file %>"/> + <target port="0"/> +</serial> -- 2.23.0

1 0

[PATCH compass-ci 2/2] mail-robot: optimize assign-account-fail-email
by Luan Shengde 27 Jan '21

27 Jan '21

add message for newly added error types. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 242dfe1..bcc4326 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -33,17 +33,20 @@ def email_err_message(message) https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… EMAIL_MESSAGE - when 'COMMIT_URL_NOT_AVAILABLE' + when 'COMMIT_AUTHOR_ERROR' err_message = <<~EMAIL_MESSAGE - We cannot confirm the commit url matches your email. + We cannot confirm the commit author matches your email. + Make sure it is truely submitted with your email. - Make sure the commit exists. - Make sure it is truely submitted with your email. - - Go to the commit history and find your commit. - Enter the commit and you will get your commit url. + Get a commit than was submitted with your own email and try again. + EMAIL_MESSAGE + when 'NO_COMMIT_ID' + err_message = <<~EMAIL_MESSAGE + We can not find such commit ID from the repo logs - Try again after you have correctted the url + Please check the commit ID exists. + or + You have write the commit id correctly. EMAIL_MESSAGE when 'NO_PUBKEY' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: fix bug for parsing email
by Luan Shengde 27 Jan '21

27 Jan '21

[why] for non-gitee repos: we may sometimes can not get the author email from the http request feedback when checking the commit. so users may still receive a fail email even if they send 'apply account' email correctly. [how] both gitee and non-gitee will clone the repo tree and check the commit: - check commit id exists - check commit author matches the email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 42 ++++++++++--------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index a41a89a..95b967a 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -6,7 +6,6 @@ require 'json' require 'mail' require 'rest-client' -require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey # be called by AssignAccount when it needs to extract required data: @@ -91,29 +90,34 @@ class ParseApplyAccountEmail end def commit_url_availability(url, base_url) - hub_name = url.split('/')[2] - - # it requires authentication when sending request to get the commit information - # clone the repo and then validate the commit for the email address - if hub_name.eql? 'gitee.com' - gitee_commit(url, base_url) - else - non_gitee_commit(url) - end - end + repo_url = [base_url, 'git'].join('.') + repo_dir = repo_url.split('/')[-1] + commit_id = url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone --bare #{repo_url} #{repo_dir}) + + # get all commit IDs and check commit id exists + repo_commits = %x(git -C #{repo_dir} log --pretty=format:'%H').split(/\n/) + check_commit_exist(commit_id, repo_commits) - def gitee_commit(url, base_url) - my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) - my_gitee_commit.gitee_commit_check + # get the auther's email for the commit + author_email = %x(/usr/bin/git -C #{repo_dir} show -s --format=%aE #{commit_id}).chomp + check_commit_email(author_email) + + FileUtils.rm_rf repo_dir end - def non_gitee_commit(url) - url_fdback = RestClient.get(url).body - email_index = url_fdback.index @my_info['my_email'] + def check_commit_exist(commit_id, repo_commits) + return if repo_commits.include? commit_id + + raise 'NO_COMMIT_ID' + end - return unless email_index.nil? + def check_commit_email(author_email) + return if author_email.eql? @my_info['my_email'] - raise 'COMMIT_URL_NOT_AVAILABLE' + raise 'COMMIT_AUTHOR_ERROR' end def parse_pub_key -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies: replace gem sources
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] function linux_dep() and mac_dep() all use 'gem install', so replace gem sources in function entry run(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6b418e5f4..32d71ccf8 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -39,6 +39,7 @@ mac_dep() run() { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ if is_system "Linux"; then linux_dep elif is_system "Darwin"; then -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: modify the incorrect link
by Li Ping 27 Jan '21

27 Jan '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- sparrow/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 1906f10..703e8f9 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -78,11 +78,11 @@ ``` compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 -4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） +4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.160.11.58:11300/os/)下获取对应版本的cgz文件） ```bash mkdir -p /srv/os/openeuler/aarch64/20.03 cd /srv/os/openeuler/aarch64/20.03 - wget http://124.90.34.227:11300/os/openeuler/aarch64/20.03.cgz + wget http://api.compass-ci.openeuler.org:11300/os/openeuler/aarch64/20.03.cgz ``` 5. 解压rootfs cgz 文件 @@ -109,7 +109,7 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… ```bash cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 cat output -- 2.23.0

2 3

[PATCH v2 compass-ci] providers/lib: introduce context.rb
by Xiao Shenwei 27 Jan '21

27 Jan '21

[why] expand erb templates need a class to save context [how] Context class provider a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] update email address to apply account in README
by Li Yuanchao 27 Jan '21

27 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **登录测试环境** -1. 向 compass-ci-reply(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… +1. 向 compass-ci-robot(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… 2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… 3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… -- 2.23.0

1 0