- Compass-ci - mailweb.openeuler.org

[PATCH lab-z9] add host-file of a7 and a20
by Zhang Dewan 27 Jan '21

27 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a20 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a7 | 25 +++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a20 create mode 100644 hosts/taishan200-2280-2s64p-256g--a7 diff --git a/hosts/taishan200-2280-2s48p-256g--a20 b/hosts/taishan200-2280-2s48p-256g--a20 new file mode 100644 index 0000000..21bf8e1 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a20 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b888f2dd + - /dev/disk/by-id/scsi-350000399b888f705 + - /dev/disk/by-id/scsi-350000399b88906f5 + - /dev/disk/by-id/scsi-350000399b8890141 +mac_addr: + - 84:46:fe:3d:a7:d5 + - 84:46:fe:3d:a7:d6 + - 84:46:fe:3d:a7:d7 + - 84:46:fe:3d:a7:d8 + - 84:46:fe:3d:a7:d9 + - 84:46:fe:3d:a7:da + - 84:46:fe:3d:a7:db + - 84:46:fe:3d:a7:dc +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.10 diff --git a/hosts/taishan200-2280-2s64p-256g--a7 b/hosts/taishan200-2280-2s64p-256g--a7 new file mode 100644 index 0000000..07701d3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a7 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039998134611 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43104 +mac_addr: + - 84:46:fe:89:be:20 + - 84:46:fe:89:be:21 + - 84:46:fe:89:be:22 + - 84:46:fe:89:be:23 + - ac:b3:b5:1e:d6:eb + - ac:b3:b5:1e:d6:ec + - ac:b3:b5:1e:d6:ed + - ac:b3:b5:1e:d6:ee + - ac:b3:b5:1e:d7:7b + - ac:b3:b5:1e:d7:7c + - ac:b3:b5:1e:d7:7d + - ac:b3:b5:1e:d7:7e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.7 -- 2.23.0

1 0

[PATCH compass-ci] lib/build_my_info_client: update my_info before store into ES
by Liu Yinsi 27 Jan '21

27 Jan '21

before: when store account info to ES, generate new my_token. after: when store account info to ES, if my_email exist, use old my_token, if not exist, generate new my_token. [why] prevent my_token in config yaml is different with ES. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/build_my_info_client.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) mode change 100644 => 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb old mode 100644 new mode 100755 index c946bec..6aa842c --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -66,12 +66,16 @@ class BuildMyInfo def store_account_info es = ESClient.new(index: 'accounts') + + my_info_es = es.query_by_id(@my_info['my_email']) || {} + @my_info.update my_info_es + es.put_source_by_id(@my_info['my_email'], @my_info) end def config_my_info + store_account_info config_default_yaml config_lab_yaml - store_account_info end end -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 27 Jan '21

27 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..91c659d 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,26 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 1

[PATCH compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 27 Jan '21

27 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..2532988 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,25 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 2

[PATCH lkp-tests] stats: add output failed msg
by Zhang Yale 27 Jan '21

27 Jan '21

[Why] Output meed to be displayed in the ansible_test.json file. [Input] fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined\n\nThe error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Define mysql_packages.\n ^ here\n"} PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 [Output] playbook.run.msg: The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined The error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Define mysql_packages. ^ here nr_ok: 2 nr_changed: 0 nr_failed: 1 nr_skipped: 1 nr_rescued: 0 nr_ignored: 0 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3..f0766412 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,17 @@ #!/usr/bin/env ruby +require 'json' while (line = STDIN.gets) case line.chomp! + when /FAILED!\s+=>\s+(\{.*\})/ + value = JSON.load($1)['msg'] + puts "playbook.run.msg: #{value}" when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + puts "nr_ok: #{$1}" + puts "nr_changed: #{$2}" + puts "nr_failed: #{$4}" + puts "nr_skipped: #{$5}" + puts "nr_rescued: #{$6}" + puts "nr_ignored: #{$7}" end end -- 2.23.0

2 2

[PATCH compass-ci] sbin/build-my-info: optimize options
by Luan Shengde 27 Jan '21

27 Jan '21

add output usage add option -h|--help set default option to -h|--help Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..028fbac 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -14,6 +14,11 @@ option = { } options = OptionParser.new do |opts| + opts.banner = "Usage: ./build-my-info.rb -e email -n name -l lab\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| option[:my_email] = email end @@ -25,8 +30,15 @@ options = OptionParser.new do |opts| opts.on('-l lab', 'lab') do |lab| option[:lab] = lab end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end end +ARGV << '-h' if ARGV.empty? + options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -- 2.23.0

1 0

[PATCH v1 compass-ci 2/2] providers/libvirt: split common elements of domain.xml
by Xiao Shenwei 27 Jan '21

27 Jan '21

[why] some elements can be specifies by user. case 1: a complete domain.xml we should replace some elements of user's xml like os name ... case 2: default domain.xml generate a default domain.xml case 3: only specifies a cpu templates use user's cpu template replace default cpu template case 4: composition template user can specifies different tempaltes like cpu, disk, ... [how] split domain.xml then template combination based on user requirements Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/templates/active.xml | 3 +++ providers/libvirt/templates/clock.xml | 6 ++++++ providers/libvirt/templates/cpu.xml | 6 ++++++ providers/libvirt/templates/devices.xml | 3 +++ providers/libvirt/templates/disk.xml | 7 +++++++ providers/libvirt/templates/domain.xml | 10 ++++++++++ providers/libvirt/templates/emulator.xml | 1 + providers/libvirt/templates/interface.xml | 6 ++++++ providers/libvirt/templates/memory.xml | 1 + providers/libvirt/templates/name.xml | 1 + providers/libvirt/templates/os.xml | 8 ++++++++ providers/libvirt/templates/seclabel.xml | 4 ++++ providers/libvirt/templates/serial.xml | 4 ++++ 13 files changed, 60 insertions(+) create mode 100644 providers/libvirt/templates/active.xml create mode 100644 providers/libvirt/templates/clock.xml create mode 100644 providers/libvirt/templates/cpu.xml create mode 100644 providers/libvirt/templates/devices.xml create mode 100644 providers/libvirt/templates/disk.xml create mode 100644 providers/libvirt/templates/domain.xml create mode 100644 providers/libvirt/templates/emulator.xml create mode 100644 providers/libvirt/templates/interface.xml create mode 100644 providers/libvirt/templates/memory.xml create mode 100644 providers/libvirt/templates/name.xml create mode 100644 providers/libvirt/templates/os.xml create mode 100644 providers/libvirt/templates/seclabel.xml create mode 100644 providers/libvirt/templates/serial.xml diff --git a/providers/libvirt/templates/active.xml b/providers/libvirt/templates/active.xml new file mode 100644 index 0000000..8ed7ee4 --- /dev/null +++ b/providers/libvirt/templates/active.xml @@ -0,0 +1,3 @@ +<on_poweroff>destroy</on_poweroff> +<on_reboot>destroy</on_reboot> +<on_crash>destroy</on_crash> diff --git a/providers/libvirt/templates/clock.xml b/providers/libvirt/templates/clock.xml new file mode 100644 index 0000000..7646c91 --- /dev/null +++ b/providers/libvirt/templates/clock.xml @@ -0,0 +1,6 @@ +<clock offset='utc'> + <timer name='rtc' tickpolicy='catchup' track='guest'/> + <timer name='hpet' present='no'/> + <timer name='pit' tickpolicy='delay'/> +</clock> + diff --git a/providers/libvirt/templates/cpu.xml b/providers/libvirt/templates/cpu.xml new file mode 100644 index 0000000..6991457 --- /dev/null +++ b/providers/libvirt/templates/cpu.xml @@ -0,0 +1,6 @@ +<vcpu placement='static'><%= nr_cpu %></vcpu> +<cpu mode='host-passthrough' check='none'> +  +</cpu> diff --git a/providers/libvirt/templates/devices.xml b/providers/libvirt/templates/devices.xml new file mode 100644 index 0000000..83cd847 --- /dev/null +++ b/providers/libvirt/templates/devices.xml @@ -0,0 +1,3 @@ +<devices> + <graphics type='vnc' autoport='yes' listen='0.0.0.0'><graphics> +</devices> diff --git a/providers/libvirt/templates/disk.xml b/providers/libvirt/templates/disk.xml new file mode 100644 index 0000000..7aecef0 --- /dev/null +++ b/providers/libvirt/templates/disk.xml @@ -0,0 +1,7 @@ +<% if disk %> +<disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='<%= disk %>'/> + <target dev='vda' bus='virtio'/> +</disk> +<% end %> diff --git a/providers/libvirt/templates/domain.xml b/providers/libvirt/templates/domain.xml new file mode 100644 index 0000000..a3fdb63 --- /dev/null +++ b/providers/libvirt/templates/domain.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" character="utf-8"?> +<domain type='kvm'> + <resource> + <partition>/machine</partition> + </resource> + <features> + <acpi/> + <gic version='3'/> + </features> +</domain> diff --git a/providers/libvirt/templates/emulator.xml b/providers/libvirt/templates/emulator.xml new file mode 100644 index 0000000..e915215 --- /dev/null +++ b/providers/libvirt/templates/emulator.xml @@ -0,0 +1 @@ +<emulator><%=qemu_path%></emulator> diff --git a/providers/libvirt/templates/interface.xml b/providers/libvirt/templates/interface.xml new file mode 100644 index 0000000..5d4c04f --- /dev/null +++ b/providers/libvirt/templates/interface.xml @@ -0,0 +1,6 @@ +<interface type='bridge'> + <source bridge='br0'/> + <mac address='<%= mac %>'/> + <target dev='vnet0'/> + <model type='virtio'/> +</interface> diff --git a/providers/libvirt/templates/memory.xml b/providers/libvirt/templates/memory.xml new file mode 100644 index 0000000..34aec49 --- /dev/null +++ b/providers/libvirt/templates/memory.xml @@ -0,0 +1 @@ +<memory unit='GiB'><%= memory.to_i %></memory> diff --git a/providers/libvirt/templates/name.xml b/providers/libvirt/templates/name.xml new file mode 100644 index 0000000..5dc98f4 --- /dev/null +++ b/providers/libvirt/templates/name.xml @@ -0,0 +1 @@ +<name><%= job_id %></name> diff --git a/providers/libvirt/templates/os.xml b/providers/libvirt/templates/os.xml new file mode 100644 index 0000000..317866d --- /dev/null +++ b/providers/libvirt/templates/os.xml @@ -0,0 +1,8 @@ +<os> + <type arch='<%= arch %>' machine='virt'>hvm</type> + <loader readonly='yes' type='pflash'><%= loader %></loader> + <kernel><%= kernel %></kernel> + <initrd><%= initrd %></initrd> + <cmdline><%= cmdline %></cmdline> + <nvram>/var/lib/libvirt/qemu/nvram/<%= job_id %>.fd</nvram> +</os> diff --git a/providers/libvirt/templates/seclabel.xml b/providers/libvirt/templates/seclabel.xml new file mode 100644 index 0000000..5c10a77 --- /dev/null +++ b/providers/libvirt/templates/seclabel.xml @@ -0,0 +1,4 @@ +<seclabel type='dynamic' model='dac' relabel='yes'> + <label>+0:+0</label> + <imagelabel>+0:+0</imagelabel> +</seclabel> diff --git a/providers/libvirt/templates/serial.xml b/providers/libvirt/templates/serial.xml new file mode 100644 index 0000000..2edc882 --- /dev/null +++ b/providers/libvirt/templates/serial.xml @@ -0,0 +1,4 @@ +<serial type='file'> + <source path="<%= log_file %>"/> + <target port="0"/> +</serial> -- 2.23.0

1 0

[PATCH compass-ci 2/2] mail-robot: optimize assign-account-fail-email
by Luan Shengde 27 Jan '21

27 Jan '21

add message for newly added error types. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 242dfe1..bcc4326 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -33,17 +33,20 @@ def email_err_message(message) https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… EMAIL_MESSAGE - when 'COMMIT_URL_NOT_AVAILABLE' + when 'COMMIT_AUTHOR_ERROR' err_message = <<~EMAIL_MESSAGE - We cannot confirm the commit url matches your email. + We cannot confirm the commit author matches your email. + Make sure it is truely submitted with your email. - Make sure the commit exists. - Make sure it is truely submitted with your email. - - Go to the commit history and find your commit. - Enter the commit and you will get your commit url. + Get a commit than was submitted with your own email and try again. + EMAIL_MESSAGE + when 'NO_COMMIT_ID' + err_message = <<~EMAIL_MESSAGE + We can not find such commit ID from the repo logs - Try again after you have correctted the url + Please check the commit ID exists. + or + You have write the commit id correctly. EMAIL_MESSAGE when 'NO_PUBKEY' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: fix bug for parsing email
by Luan Shengde 27 Jan '21

27 Jan '21

[why] for non-gitee repos: we may sometimes can not get the author email from the http request feedback when checking the commit. so users may still receive a fail email even if they send 'apply account' email correctly. [how] both gitee and non-gitee will clone the repo tree and check the commit: - check commit id exists - check commit author matches the email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 42 ++++++++++--------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index a41a89a..95b967a 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -6,7 +6,6 @@ require 'json' require 'mail' require 'rest-client' -require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey # be called by AssignAccount when it needs to extract required data: @@ -91,29 +90,34 @@ class ParseApplyAccountEmail end def commit_url_availability(url, base_url) - hub_name = url.split('/')[2] - - # it requires authentication when sending request to get the commit information - # clone the repo and then validate the commit for the email address - if hub_name.eql? 'gitee.com' - gitee_commit(url, base_url) - else - non_gitee_commit(url) - end - end + repo_url = [base_url, 'git'].join('.') + repo_dir = repo_url.split('/')[-1] + commit_id = url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone --bare #{repo_url} #{repo_dir}) + + # get all commit IDs and check commit id exists + repo_commits = %x(git -C #{repo_dir} log --pretty=format:'%H').split(/\n/) + check_commit_exist(commit_id, repo_commits) - def gitee_commit(url, base_url) - my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) - my_gitee_commit.gitee_commit_check + # get the auther's email for the commit + author_email = %x(/usr/bin/git -C #{repo_dir} show -s --format=%aE #{commit_id}).chomp + check_commit_email(author_email) + + FileUtils.rm_rf repo_dir end - def non_gitee_commit(url) - url_fdback = RestClient.get(url).body - email_index = url_fdback.index @my_info['my_email'] + def check_commit_exist(commit_id, repo_commits) + return if repo_commits.include? commit_id + + raise 'NO_COMMIT_ID' + end - return unless email_index.nil? + def check_commit_email(author_email) + return if author_email.eql? @my_info['my_email'] - raise 'COMMIT_URL_NOT_AVAILABLE' + raise 'COMMIT_AUTHOR_ERROR' end def parse_pub_key -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies: replace gem sources
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] function linux_dep() and mac_dep() all use 'gem install', so replace gem sources in function entry run(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6b418e5f4..32d71ccf8 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -39,6 +39,7 @@ mac_dep() run() { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ if is_system "Linux"; then linux_dep elif is_system "Darwin"; then -- 2.23.0

1 0