- Compass-ci - mailweb.openeuler.org

[PATCH lab-z9] add host-file of a7 and a20
by Zhang Dewan 27 Jan '21

27 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a20 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a7 | 25 +++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a20 create mode 100644 hosts/taishan200-2280-2s64p-256g--a7 diff --git a/hosts/taishan200-2280-2s48p-256g--a20 b/hosts/taishan200-2280-2s48p-256g--a20 new file mode 100644 index 0000000..21bf8e1 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a20 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b888f2dd + - /dev/disk/by-id/scsi-350000399b888f705 + - /dev/disk/by-id/scsi-350000399b88906f5 + - /dev/disk/by-id/scsi-350000399b8890141 +mac_addr: + - 84:46:fe:3d:a7:d5 + - 84:46:fe:3d:a7:d6 + - 84:46:fe:3d:a7:d7 + - 84:46:fe:3d:a7:d8 + - 84:46:fe:3d:a7:d9 + - 84:46:fe:3d:a7:da + - 84:46:fe:3d:a7:db + - 84:46:fe:3d:a7:dc +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.10 diff --git a/hosts/taishan200-2280-2s64p-256g--a7 b/hosts/taishan200-2280-2s64p-256g--a7 new file mode 100644 index 0000000..07701d3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a7 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039998134611 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43104 +mac_addr: + - 84:46:fe:89:be:20 + - 84:46:fe:89:be:21 + - 84:46:fe:89:be:22 + - 84:46:fe:89:be:23 + - ac:b3:b5:1e:d6:eb + - ac:b3:b5:1e:d6:ec + - ac:b3:b5:1e:d6:ed + - ac:b3:b5:1e:d6:ee + - ac:b3:b5:1e:d7:7b + - ac:b3:b5:1e:d7:7c + - ac:b3:b5:1e:d7:7d + - ac:b3:b5:1e:d7:7e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.7 -- 2.23.0

1 0

[PATCH compass-ci] lib/build_my_info_client: update my_info before store into ES
by Liu Yinsi 27 Jan '21

27 Jan '21

before: when store account info to ES, generate new my_token. after: when store account info to ES, if my_email exist, use old my_token, if not exist, generate new my_token. [why] prevent my_token in config yaml is different with ES. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/build_my_info_client.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) mode change 100644 => 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb old mode 100644 new mode 100755 index c946bec..6aa842c --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -66,12 +66,16 @@ class BuildMyInfo def store_account_info es = ESClient.new(index: 'accounts') + + my_info_es = es.query_by_id(@my_info['my_email']) || {} + @my_info.update my_info_es + es.put_source_by_id(@my_info['my_email'], @my_info) end def config_my_info + store_account_info config_default_yaml config_lab_yaml - store_account_info end end -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 27 Jan '21

27 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..91c659d 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,26 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 1

[PATCH compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 27 Jan '21

27 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..2532988 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,25 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 2

[PATCH lkp-tests] stats: add output failed msg
by Zhang Yale 27 Jan '21

27 Jan '21

[Why] Output meed to be displayed in the ansible_test.json file. [Input] fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined\n\nThe error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Define mysql_packages.\n ^ here\n"} PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 [Output] playbook.run.msg: The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined The error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Define mysql_packages. ^ here nr_ok: 2 nr_changed: 0 nr_failed: 1 nr_skipped: 1 nr_rescued: 0 nr_ignored: 0 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3..f0766412 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,17 @@ #!/usr/bin/env ruby +require 'json' while (line = STDIN.gets) case line.chomp! + when /FAILED!\s+=>\s+(\{.*\})/ + value = JSON.load($1)['msg'] + puts "playbook.run.msg: #{value}" when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + puts "nr_ok: #{$1}" + puts "nr_changed: #{$2}" + puts "nr_failed: #{$4}" + puts "nr_skipped: #{$5}" + puts "nr_rescued: #{$6}" + puts "nr_ignored: #{$7}" end end -- 2.23.0

2 2

[PATCH compass-ci] sbin/build-my-info: optimize options
by Luan Shengde 27 Jan '21

27 Jan '21

add output usage add option -h|--help set default option to -h|--help Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..028fbac 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -14,6 +14,11 @@ option = { } options = OptionParser.new do |opts| + opts.banner = "Usage: ./build-my-info.rb -e email -n name -l lab\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| option[:my_email] = email end @@ -25,8 +30,15 @@ options = OptionParser.new do |opts| opts.on('-l lab', 'lab') do |lab| option[:lab] = lab end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end end +ARGV << '-h' if ARGV.empty? + options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -- 2.23.0

1 0

[PATCH v1 compass-ci 2/2] providers/libvirt: split common elements of domain.xml
by Xiao Shenwei 27 Jan '21

27 Jan '21

[why] some elements can be specifies by user. case 1: a complete domain.xml we should replace some elements of user's xml like os name ... case 2: default domain.xml generate a default domain.xml case 3: only specifies a cpu templates use user's cpu template replace default cpu template case 4: composition template user can specifies different tempaltes like cpu, disk, ... [how] split domain.xml then template combination based on user requirements Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/templates/active.xml | 3 +++ providers/libvirt/templates/clock.xml | 6 ++++++ providers/libvirt/templates/cpu.xml | 6 ++++++ providers/libvirt/templates/devices.xml | 3 +++ providers/libvirt/templates/disk.xml | 7 +++++++ providers/libvirt/templates/domain.xml | 10 ++++++++++ providers/libvirt/templates/emulator.xml | 1 + providers/libvirt/templates/interface.xml | 6 ++++++ providers/libvirt/templates/memory.xml | 1 + providers/libvirt/templates/name.xml | 1 + providers/libvirt/templates/os.xml | 8 ++++++++ providers/libvirt/templates/seclabel.xml | 4 ++++ providers/libvirt/templates/serial.xml | 4 ++++ 13 files changed, 60 insertions(+) create mode 100644 providers/libvirt/templates/active.xml create mode 100644 providers/libvirt/templates/clock.xml create mode 100644 providers/libvirt/templates/cpu.xml create mode 100644 providers/libvirt/templates/devices.xml create mode 100644 providers/libvirt/templates/disk.xml create mode 100644 providers/libvirt/templates/domain.xml create mode 100644 providers/libvirt/templates/emulator.xml create mode 100644 providers/libvirt/templates/interface.xml create mode 100644 providers/libvirt/templates/memory.xml create mode 100644 providers/libvirt/templates/name.xml create mode 100644 providers/libvirt/templates/os.xml create mode 100644 providers/libvirt/templates/seclabel.xml create mode 100644 providers/libvirt/templates/serial.xml diff --git a/providers/libvirt/templates/active.xml b/providers/libvirt/templates/active.xml new file mode 100644 index 0000000..8ed7ee4 --- /dev/null +++ b/providers/libvirt/templates/active.xml @@ -0,0 +1,3 @@ +<on_poweroff>destroy</on_poweroff> +<on_reboot>destroy</on_reboot> +<on_crash>destroy</on_crash> diff --git a/providers/libvirt/templates/clock.xml b/providers/libvirt/templates/clock.xml new file mode 100644 index 0000000..7646c91 --- /dev/null +++ b/providers/libvirt/templates/clock.xml @@ -0,0 +1,6 @@ +<clock offset='utc'> + <timer name='rtc' tickpolicy='catchup' track='guest'/> + <timer name='hpet' present='no'/> + <timer name='pit' tickpolicy='delay'/> +</clock> + diff --git a/providers/libvirt/templates/cpu.xml b/providers/libvirt/templates/cpu.xml new file mode 100644 index 0000000..6991457 --- /dev/null +++ b/providers/libvirt/templates/cpu.xml @@ -0,0 +1,6 @@ +<vcpu placement='static'><%= nr_cpu %></vcpu> +<cpu mode='host-passthrough' check='none'> +  +</cpu> diff --git a/providers/libvirt/templates/devices.xml b/providers/libvirt/templates/devices.xml new file mode 100644 index 0000000..83cd847 --- /dev/null +++ b/providers/libvirt/templates/devices.xml @@ -0,0 +1,3 @@ +<devices> + <graphics type='vnc' autoport='yes' listen='0.0.0.0'><graphics> +</devices> diff --git a/providers/libvirt/templates/disk.xml b/providers/libvirt/templates/disk.xml new file mode 100644 index 0000000..7aecef0 --- /dev/null +++ b/providers/libvirt/templates/disk.xml @@ -0,0 +1,7 @@ +<% if disk %> +<disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='<%= disk %>'/> + <target dev='vda' bus='virtio'/> +</disk> +<% end %> diff --git a/providers/libvirt/templates/domain.xml b/providers/libvirt/templates/domain.xml new file mode 100644 index 0000000..a3fdb63 --- /dev/null +++ b/providers/libvirt/templates/domain.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" character="utf-8"?> +<domain type='kvm'> + <resource> + <partition>/machine</partition> + </resource> + <features> + <acpi/> + <gic version='3'/> + </features> +</domain> diff --git a/providers/libvirt/templates/emulator.xml b/providers/libvirt/templates/emulator.xml new file mode 100644 index 0000000..e915215 --- /dev/null +++ b/providers/libvirt/templates/emulator.xml @@ -0,0 +1 @@ +<emulator><%=qemu_path%></emulator> diff --git a/providers/libvirt/templates/interface.xml b/providers/libvirt/templates/interface.xml new file mode 100644 index 0000000..5d4c04f --- /dev/null +++ b/providers/libvirt/templates/interface.xml @@ -0,0 +1,6 @@ +<interface type='bridge'> + <source bridge='br0'/> + <mac address='<%= mac %>'/> + <target dev='vnet0'/> + <model type='virtio'/> +</interface> diff --git a/providers/libvirt/templates/memory.xml b/providers/libvirt/templates/memory.xml new file mode 100644 index 0000000..34aec49 --- /dev/null +++ b/providers/libvirt/templates/memory.xml @@ -0,0 +1 @@ +<memory unit='GiB'><%= memory.to_i %></memory> diff --git a/providers/libvirt/templates/name.xml b/providers/libvirt/templates/name.xml new file mode 100644 index 0000000..5dc98f4 --- /dev/null +++ b/providers/libvirt/templates/name.xml @@ -0,0 +1 @@ +<name><%= job_id %></name> diff --git a/providers/libvirt/templates/os.xml b/providers/libvirt/templates/os.xml new file mode 100644 index 0000000..317866d --- /dev/null +++ b/providers/libvirt/templates/os.xml @@ -0,0 +1,8 @@ +<os> + <type arch='<%= arch %>' machine='virt'>hvm</type> + <loader readonly='yes' type='pflash'><%= loader %></loader> + <kernel><%= kernel %></kernel> + <initrd><%= initrd %></initrd> + <cmdline><%= cmdline %></cmdline> + <nvram>/var/lib/libvirt/qemu/nvram/<%= job_id %>.fd</nvram> +</os> diff --git a/providers/libvirt/templates/seclabel.xml b/providers/libvirt/templates/seclabel.xml new file mode 100644 index 0000000..5c10a77 --- /dev/null +++ b/providers/libvirt/templates/seclabel.xml @@ -0,0 +1,4 @@ +<seclabel type='dynamic' model='dac' relabel='yes'> + <label>+0:+0</label> + <imagelabel>+0:+0</imagelabel> +</seclabel> diff --git a/providers/libvirt/templates/serial.xml b/providers/libvirt/templates/serial.xml new file mode 100644 index 0000000..2edc882 --- /dev/null +++ b/providers/libvirt/templates/serial.xml @@ -0,0 +1,4 @@ +<serial type='file'> + <source path="<%= log_file %>"/> + <target port="0"/> +</serial> -- 2.23.0

1 0

[PATCH compass-ci 2/2] mail-robot: optimize assign-account-fail-email
by Luan Shengde 27 Jan '21

27 Jan '21

add message for newly added error types. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 242dfe1..bcc4326 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -33,17 +33,20 @@ def email_err_message(message) https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… EMAIL_MESSAGE - when 'COMMIT_URL_NOT_AVAILABLE' + when 'COMMIT_AUTHOR_ERROR' err_message = <<~EMAIL_MESSAGE - We cannot confirm the commit url matches your email. + We cannot confirm the commit author matches your email. + Make sure it is truely submitted with your email. - Make sure the commit exists. - Make sure it is truely submitted with your email. - - Go to the commit history and find your commit. - Enter the commit and you will get your commit url. + Get a commit than was submitted with your own email and try again. + EMAIL_MESSAGE + when 'NO_COMMIT_ID' + err_message = <<~EMAIL_MESSAGE + We can not find such commit ID from the repo logs - Try again after you have correctted the url + Please check the commit ID exists. + or + You have write the commit id correctly. EMAIL_MESSAGE when 'NO_PUBKEY' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: fix bug for parsing email
by Luan Shengde 27 Jan '21

27 Jan '21

[why] for non-gitee repos: we may sometimes can not get the author email from the http request feedback when checking the commit. so users may still receive a fail email even if they send 'apply account' email correctly. [how] both gitee and non-gitee will clone the repo tree and check the commit: - check commit id exists - check commit author matches the email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 42 ++++++++++--------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index a41a89a..95b967a 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -6,7 +6,6 @@ require 'json' require 'mail' require 'rest-client' -require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey # be called by AssignAccount when it needs to extract required data: @@ -91,29 +90,34 @@ class ParseApplyAccountEmail end def commit_url_availability(url, base_url) - hub_name = url.split('/')[2] - - # it requires authentication when sending request to get the commit information - # clone the repo and then validate the commit for the email address - if hub_name.eql? 'gitee.com' - gitee_commit(url, base_url) - else - non_gitee_commit(url) - end - end + repo_url = [base_url, 'git'].join('.') + repo_dir = repo_url.split('/')[-1] + commit_id = url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone --bare #{repo_url} #{repo_dir}) + + # get all commit IDs and check commit id exists + repo_commits = %x(git -C #{repo_dir} log --pretty=format:'%H').split(/\n/) + check_commit_exist(commit_id, repo_commits) - def gitee_commit(url, base_url) - my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) - my_gitee_commit.gitee_commit_check + # get the auther's email for the commit + author_email = %x(/usr/bin/git -C #{repo_dir} show -s --format=%aE #{commit_id}).chomp + check_commit_email(author_email) + + FileUtils.rm_rf repo_dir end - def non_gitee_commit(url) - url_fdback = RestClient.get(url).body - email_index = url_fdback.index @my_info['my_email'] + def check_commit_exist(commit_id, repo_commits) + return if repo_commits.include? commit_id + + raise 'NO_COMMIT_ID' + end - return unless email_index.nil? + def check_commit_email(author_email) + return if author_email.eql? @my_info['my_email'] - raise 'COMMIT_URL_NOT_AVAILABLE' + raise 'COMMIT_AUTHOR_ERROR' end def parse_pub_key -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies: replace gem sources
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] function linux_dep() and mac_dep() all use 'gem install', so replace gem sources in function entry run(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6b418e5f4..32d71ccf8 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -39,6 +39,7 @@ mac_dep() run() { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ if is_system "Linux"; then linux_dep elif is_system "Darwin"; then -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: modify the incorrect link
by Li Ping 27 Jan '21

27 Jan '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- sparrow/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 1906f10..703e8f9 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -78,11 +78,11 @@ ``` compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 -4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） +4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.160.11.58:11300/os/)下获取对应版本的cgz文件） ```bash mkdir -p /srv/os/openeuler/aarch64/20.03 cd /srv/os/openeuler/aarch64/20.03 - wget http://124.90.34.227:11300/os/openeuler/aarch64/20.03.cgz + wget http://api.compass-ci.openeuler.org:11300/os/openeuler/aarch64/20.03.cgz ``` 5. 解压rootfs cgz 文件 @@ -109,7 +109,7 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… ```bash cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 cat output -- 2.23.0

2 3

[PATCH v2 compass-ci] providers/lib: introduce context.rb
by Xiao Shenwei 27 Jan '21

27 Jan '21

[why] expand erb templates need a class to save context [how] Context class provider a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] update email address to apply account in README
by Li Yuanchao 27 Jan '21

27 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **登录测试环境** -1. 向 compass-ci-reply(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… +1. 向 compass-ci-robot(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… 2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… 3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… -- 2.23.0

1 0

[PATCH lkp-tests] pkg/mysql/PKGBUILD: fix mysql pkgbuild
by Wei Jihui 27 Jan '21

27 Jan '21

[error] CMake Error at plugin/group_replication/libmysqlgcs/rpcgen.cmake:100(MESSAGE): Could not find rpcgen ... -- Configuring incomplete, errors occurred! [how] make and make install rpcgen Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/mysql/PKGBUILD | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD index 8869e62b..0f7d8bc8 100644 --- a/pkg/mysql/PKGBUILD +++ b/pkg/mysql/PKGBUILD @@ -7,8 +7,9 @@ license=('GPLv2') source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" "0000-mysql-add-fstack-protector-strong.patch" "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "https://github.com/thkukuk/rpcsvc-proto/releases/download/v1.4/rpcsvc-proto…" "my.cnf") -md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' '7bb0372ec4e9281813a9afc67c6cfc20') +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '7bb0372ec4e9281813a9afc67c6cfc20') prepare() { @@ -19,9 +20,18 @@ prepare() build() { build_getcpu + build_rpcsvc build_mysql } +build_rpcsvc() +{ + cd "$srcdir/rpcsvc-proto-1.4" + ./configure + make + make install +} + build_mysql() { cd $srcdir/$pkgname-$pkgver -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix.rb: fix NoMethodError
by Lu Weitao 27 Jan '21

27 Jan '21

[Why] /home/wangyong/compass-ci/lib/matrix2.rb:21:in `set_pre_value': undefined method `sum' for 1:Integer (NoMethodError) Did you mean? succ root cause: if a job has *.yaml(except job.yaml) in result_root, will meet this error when merge it to stats.json directly. [How] remove the logical handle *.yaml from lib/matrix.rb at first, we should only handle $script.yaml, and the feature will be done in other patch Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 8444334..4dd03b4 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -39,21 +39,19 @@ def file_check(file) File.basename(file, '.json') when /\.json\.gz$/ File.basename(file, '.json.gz') - when /\.yaml$/ - File.basename(file, '.yaml') end end def create_stats(result_root) stats = {} - monitor_files = Dir["#{result_root}/*.{json,json.gz,yaml}"] + monitor_files = Dir["#{result_root}/*.{json,json.gz}"] monitor_files.each do |file| next unless File.size?(file) monitor = file_check(file) - next if %w[stats job].include?(monitor) + next if monitor == 'stats' extract_pre_result(stats, monitor, file) end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: export V=1 to print verbose output
by Lin Jiaxin 27 Jan '21

27 Jan '21

A developer who really wants to make silent rules enabled by default will calling AM_SILENT_RULES([yes]) in configure.ac. It make compiling packages by default does not show the full compilation/link command lines, but only short "summaries", like this: [before] EPOINFO main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/ctags-cmd.o make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/libctags_a-args.o However, we need the full compilation/link command lines, like this: [after] ./misc/gen-repoinfo main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' gcc -DHAVE_CONFIG_H -I. -I. -I. -I./main -I./peg -DHAVE_PACKCC -DUSE_SYSTEM_STRNLEN -DHAVE_REPOINFO_H -D_FORTIFY_SOURCE=2 -std=gnu99 -Wall -I/usr/include/libxml2 -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -MT main/ctags-cmd.o -MD -MP -MF main/.deps/ctags-cmd.Tpo -c -o main/ctags-cmd.o `test -f 'main/cmd.c' || echo './'`main/cmd.c make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' mv -f main/.deps/ctags-cmd.Tpo main/.deps/ctags-cmd.Po make V=1 will produce verbose output, so export V=1 in build-pkg script. refer to: https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rule… https://lists.gnu.org/archive/html/bug-autoconf/2012-01/msg00009.html Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 5344da2a..f3545a19 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -157,6 +157,8 @@ build_source_pkg() local mark="true" [ "${os}" == "archlinux" ] && local mark="pacman" + export V=1 + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf -- 2.23.0

1 0

[PATCH compass-ci] update email address to apply account in README
by Li Yuanchao 27 Jan '21

27 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI �� git repos�� **��** -1. �� compass-ci-reply(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� +1. �� compass-ci-robot(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� 2. ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9C%AC%E5%9C%B0%E5%AE%89%E8%A3%85compass-ci%E5%AE%A2%E6%88%B7%E7%AB%AF.md)�� 3. �� sshd ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%99%BB%E5%BD%95%E6%B5%8B%E8%AF%95%E6%9C%BA%E8%B0%83%E6%B5%8B%E4%BB%BB%E5%8A%A1.md)�� -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs.rb: fix error if jobs do not contain stats
by Wu Zhende 27 Jan '21

27 Jan '21

[Before] Traceback (most recent call last): 3: from /home/wuzhende/compass-ci/sbin/es-jobs:55:in `<main>' 2: from /home/wuzhende/compass-ci/lib/es_jobs.rb:167:in `output' 1: from /home/wuzhende/compass-ci/lib/es_jobs.rb:137:in `query_jobs_state' /home/wuzhende/compass-ci/lib/es_jobs.rb:99:in `initialize_result_hash': undefined method `each' for nil:NilClass (NoMethodError) [After] kvcount.category=benchmark: 1 kvcount.os=openeuler: 1 kvcount.suite=iperf: 1 kvcount.tbox_group=vm-2p16g: 1 raw.id.[arch=aarch64]: ["crystal.786605"] raw.id.[category=benchmark]: ["crystal.786605"] raw.id.[os=openeuler]: ["crystal.786605"] raw.id.[suite=iperf]: ["crystal.786605"] raw.id.[tbox_group=vm-2p16g]: ["crystal.786605"] Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index eb3f70d..e367a91 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -84,7 +84,7 @@ class ESJobs metrics.concat(stats.keys) end - metrics.uniq! + metrics.uniq end def initialize_result_hash(metrics) @@ -137,11 +137,11 @@ class ESJobs metrics = get_all_metrics(jobs) result = initialize_result_hash(metrics) jobs.each do |job| + kvcount(result, job) stats = job['stats'] next unless stats set_default_value(result, stats, metrics) - kvcount(result, job) end stats_count(result) -- 2.23.0

2 1

[PATCH v5 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 27 Jan '21

27 Jan '21

Add limit for borrowing machine: borrowing period <= 10 days Only jobs that runtime/sleep value is less than or equal to 10 days are allowed to borrow machines. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..5448082 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,19 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + + # case runtime value is more than 10 days, job will fail. + raise error_msg if hash["runtime"].to_i > max_run_time + raise error_msg if hash["sleep"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 1

[PATCH lkp-tests] distro/depends: fix install sysbench pkg depends
by Wei Jihui 27 Jan '21

27 Jan '21

[error] Error: Unable to find a match: mysql-devel [why] openeuler 20.03 has not libmysqlclient-dev: mysql-devel change it to libmariadbclient-dev Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/sysbench-dev | 2 +- distro/depends/sysbench-mysql-dev | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/distro/depends/sysbench-dev b/distro/depends/sysbench-dev index 3fa67deb..f570c463 100644 --- a/distro/depends/sysbench-dev +++ b/distro/depends/sysbench-dev @@ -1,2 +1,2 @@ pkg-config -libmysqlclient-dev +libmariadbclient-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev index ca5b03d8..65633b33 100644 --- a/distro/depends/sysbench-mysql-dev +++ b/distro/depends/sysbench-mysql-dev @@ -1,3 +1,2 @@ pkg-config -libmysqlclient-dev mysql-server -- 2.23.0

2 1

[PATCH compass-ci] providers/lib: introduce context.rb
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] expand erb templates need a class to save context [how] Context class provider a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..1ea36b0 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

2 4

[PATCH lkp-tests] jobs: add multi-qemu.yaml multi-docker.yaml
by Xiao Shenwei 26 Jan '21

26 Jan '21

submit the job to a physical machine, then the machine will start some clients to consume job. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- jobs/multi-docker.yaml | 17 +++++++++++++++++ jobs/multi-qemu.yaml | 23 +++++++++++++++++++++++ 2 files changed, 40 insertions(+) create mode 100644 jobs/multi-docker.yaml create mode 100644 jobs/multi-qemu.yaml diff --git a/jobs/multi-docker.yaml b/jobs/multi-docker.yaml new file mode 100644 index 000000000..8e42208a9 --- /dev/null +++ b/jobs/multi-docker.yaml @@ -0,0 +1,17 @@ +suite: multi-docker +category: functional +simplify-ci: +runtime: 100d +multi-docker-0: + nr_container: 20 + tbox_group: dc-1g +multi-docker-1: + nr_container: 10 + tbox_group: dc-2g +multi-docker-2: + nr_container: 5 + tbox_group: dc-4g +multi-docker-3: + nr_container: 20 + tbox_group: dc-8g +sleep: diff --git a/jobs/multi-qemu.yaml b/jobs/multi-qemu.yaml new file mode 100644 index 000000000..200f66065 --- /dev/null +++ b/jobs/multi-qemu.yaml @@ -0,0 +1,23 @@ +suite: multi-qemu +category: functional +simplify-ci: +runtime: 100d +multi-qemu-0: + nr_vm: 5 + tbox_group: vm-2p1g +multi-qemu-1: + nr_vm: 5 + tbox_group: vm-2p2g +multi-qemu-2: + nr_vm: 5 + tbox_group: vm-2p4g +multi-qemu-3: + nr_vm: 10 + tbox_group: vm-2p8g +multi-qemu-4: + nr_vm: 5 + tbox_group: vm-2p16g +multi-qemu-5: + nr_vm: 1 + tbox_group: vm-2p32g +sleep: -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/libvirt: add a libvirt provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

request api: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 60 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100755 providers/libvirt/run.rb diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb new file mode 100755 index 0000000..b4f8b4c --- /dev/null +++ b/providers/libvirt/run.rb @@ -0,0 +1,60 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'digest/md5' +require_relative "#{ENV['CCI_SRC']}/lib/log" +require_relative "#{ENV['CCI_SRC']}/lib/sched_client" +require_relative "#{ENV['CCI_SRC']}/providers/lib/context" + +def create_logger(hostname) + filename = "#{hostname}.log" + File.delete(filename) if FileTest.exist?(filename) + Log.new filename +end + +def compute_mac(string) + /(..)(..)(..)(..)(..)/ =~ Digest::MD5.hexdigest(string) + "0a:#{$1}:#{$2}:#{$3}:#{$4}:#{$5}" +end + +def job_exist?(response) + flag = false + if response['job_id'].empty? + puts '----------' + puts 'No job now' + puts '----------' + flag = true + end + return flag +end + +def request_job(context, sched_client, logger) + mac = context.info['mac'] + hostname = context.info['hostname'] + queues = context.info['queues'] + sched_client.register_mac2host(hostname, mac) + sched_client.register_host2queues(hostname, queues) + response = JSON.parse(sched_client.consume_job('libvirt', 'mac', mac)) + if job_exist?(response) + logger.info('No job now') + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) + response = nil + end + return response +end + +def main(hostname, queues) + logger = create_logger(hostname) + mac = compute_mac hostname + context = Context.new(mac, hostname, queues) + sched_client = SchedClient.new + response = request_job(context, sched_client, logger) + return if response.nil? + + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) +end -- 2.23.0

1 0

[PATCH compass-ci] providers/libvirt: add a libvirt provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

request api: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 67 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100755 providers/libvirt/run.rb diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb new file mode 100755 index 0000000..0ff0d4f --- /dev/null +++ b/providers/libvirt/run.rb @@ -0,0 +1,67 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'digest/md5' +require_relative "#{ENV['CCI_SRC']}/lib/log" +require_relative "#{ENV['CCI_SRC']}/lib/sched_client" +require_relative "#{ENV['CCI_SRC']}/providers/lib/context" + +def create_logger(hostname) + filename = "#{hostname}.log" + File.delete(filename) if FileTest.exist?(filename) + Log.new filename +end + +def compute_mac(string) + /(..)(..)(..)(..)(..)/ =~ Digest::MD5.hexdigest(string) + "0a:#{$1}:#{$2}:#{$3}:#{$4}:#{$5}" +end + +def nojob?(response) + flag = false + if response['job_id'].empty? + puts '----------' + puts 'No job now' + puts '----------' + flag = true + end + return flag +end + +def request(context, sched_client, logger) + mac = context.info['mac'] + hostname = context.info['hostname'] + queues = context.info['queues'] + sched_client.register_mac2host(hostname, mac) + sched_client.register_host2queues(hostname, queues) + response = JSON.parse(sched_client.consume_job('libvirt', 'mac', mac)) + if nojob?(response) + logger.info('No job now') + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) + response = nil + end + return response +end + +def main(hostname, queues) + logger = create_logger(hostname) + mac = compute_mac hostname + context = Context.new(mac, hostname, queues) + sched_client = SchedClient.new + response = request(context, sched_client, logger) + return if response.nil? + + # TODO + # based on response load file + # TODO + # generate domain.xml + # TODO + # use ruby-libvirt api start a vm + # clean env + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) +end -- 2.23.0

3 4

[PATCH lkp-tests] tests/build-pkg: export V=1 to print verbose output
by Lin Jiaxin 26 Jan '21

26 Jan '21

A developer who really wants to make silent rules enabled by default will calling AM_SILENT_RULES([yes]) in configure.ac. It make compiling packages by default does not show the full compilation/link command lines, but only short "summaries", like this: EPOINFO main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/ctags-cmd.o make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/libctags_a-args.o However, we need the full compilation/link command lines, like this: ./misc/gen-repoinfo main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' gcc -DHAVE_CONFIG_H -I. -I. -I. -I./main -I./peg -DHAVE_PACKCC -DUSE_SYSTEM_STRNLEN -DHAVE_REPOINFO_H -D_FORTIFY_SOURCE=2 -std=gnu99 -Wall -I/usr/include/libxml2 -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -MT main/ctags-cmd.o -MD -MP -MF main/.deps/ctags-cmd.Tpo -c -o main/ctags-cmd.o `test -f 'main/cmd.c' || echo './'`main/cmd.c make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' mv -f main/.deps/ctags-cmd.Tpo main/.deps/ctags-cmd.Po make V=1 will produce verbose output, so export V=1 in build-pkg script. refer to: https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rule… https://lists.gnu.org/archive/html/bug-autoconf/2012-01/msg00009.html Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 5344da2a..f3545a19 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -157,6 +157,8 @@ build_source_pkg() local mark="true" [ "${os}" == "archlinux" ] && local mark="pacman" + export V=1 + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf -- 2.23.0

2 1

[PATCH compass-ci] doc/manual: optimize borrow-machine.en.md
by Luan Shengde 26 Jan '21

26 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/borrow-machine.en.md | 341 +++++++++----------------------- 1 file changed, 98 insertions(+), 243 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index 6a1fe19..36823c3 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -1,246 +1,101 @@ -# Prerequisites - -Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - -- Send an email to apply for an account. -- Receive an email from compass-ci-robot(a)qq.com. -- Configure the local environment. - -# Applying for a Test Machine (VM) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (VM). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170593 - query=>{"job_id":["z9.170593"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} - {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} - {"job_id": "z9.170593", "job_state": "boot"} - {"job_id": "z9.170593", "job_state": "download"} - {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - {"job_state":"running","job_id":"z9.170593"} - {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 06:44:18 CST 2020 - - System load: 0.83 - Processes: 107 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.18.156.13 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (VM) after use. - - ```shell - root@vm-2p8g ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - -# Applying for a Test Machine (Physical Machine) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (physical machine). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170594 - query=>{"job_id":["z9.170594"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} - {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} - {"job_id": "z9.170594", "job_state": "boot"} - {"job_id": "z9.170594", "job_state": "download"} - {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} - {"job_state":"running","job_id":"z9.170594"} - {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 14:51:59 CST 2020 - - System load: 1.31 - Processes: 1020 - Memory used: 5.1% - Swap used: 0.0% - Usage On: 3% - IP address: 172.168.178.48 - Users online: 1 - - - - root@taishan200-2280-2s64p-256g--a5 ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (physical machine) after use. - - ```shell - root@taishan200-2280-2s64p-256g--a5 ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` +# Prepare + +- Apply account +- Config default yaml files + +If have not completed above works, reference to [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… to finish it. + +# Apply testbox + +## 1. Generate local ssh pub key + + Use the command below to check the pub key exists: + + ls ~/.ssh/*.pub + + Generate one if you don't have one now: + + ssh-keygen + +## 2. Select the job yaml + + We have provided various examples for you in dir ~/lkp-tests/jobs: + + You can filtrate examples for borrowing machine with command below: + + cd ~/lkp-tests/jobs + ls borrow* + +## 3. Submit job + + Command to submit jobs: + + for DCs: + + submit -c -m testbox=dc-2g os_mount=container docker_image=centos:8 borrow-1h.yaml + + for VMs: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + for HWs: + + submit -c -m testbox=taishan200-2280-2s48p-256g borrow-1h.yaml + + - You can view the logs in real time for the job with the command above。 + - You will directly login the testbox if the job runs successfully. + - And you will receive an email that contains login command and server configuration information. + - Only within the period of borrowing machine, you can access the textbox with the login command. + +## 4. Return testbox + + Reruan manually(recommended): + + Manually execute 'reboot' in time to return the testbox. + Avoid a waste of computer resource with no-load running. + + Automatically return on maturity: + + The testbox will be returned automatically if it has expired its service life. + + - All testboxes will be returned if it has been executed command 'reboot'. + - The testbox cannot be accessed any more after it has been returned. + - Apply a new one if you want continue to use it. # FAQ -* How Do I Change the Duration of Keeping the Test Machine when Applying for It? - - ```shell - hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml - suite: borrow - testcase: borrow - - runtime: 1h - ssh_pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: - hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the runtime field. - hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml - # After changing the value, submit the request again. - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - ``` - -* Guide to the **submit** Command - - Reference: [submit-job.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… - -* What Are the testbox Options? - - For details about the testbox options, visit https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts. - - > ![](./../public_sys-resources/icon-note.gif) **Note** - > - > VM testbox: vm-xxx - > - > PM testbox: taishan200-2280-xxx - - > ![](./../public_sys-resources/icon-notice.gif) **Notice** - > -> - If the testbox of a physical machine ends with `--axx`, a physical machine is specified. If a task is already in the task queue of the physical machine, the borrow task you submitted will not be processed until the previous task in the queue is completed. - > - If the testbox of a physical machine does not end with `-axx`, no physical machine is specified. In this case, the borrow task you submitted will be immediately allocated to idle physical machines in the cluster for execution. - -* How Do I Borrow the Specified Operating System? - - For details about the supported `os`, `os_arch`, and `os_version`, see [os-os\_verison-os\_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/…. +## Customize the borrowing time + + Find key in the yaml file and edit its value according to your requirement. + + - The borrowing period can be calculated in days and hours. + - The maximum period is no more than 10 day. + +## Guidance for command submit + + See the usage and options for command 'submit' the command below: + + submit -h + + Reference the following line to learn the advanced usage for command 'submit': + + [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… + +## Available testbox + + For a full list of testbox, reference to https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + + >![](./../public_sys-resources/icon-note.gif) **instruction:** + > + > - DCs: dc-xxx + > - VMs: vm-xxx + > - HWs: taishan200-2280-xxx + + >![](./../public_sys-resources/icon-notice.gif) **attention:** + > - It means that you choosed a specified physical machine if the testbox name is end with `--axx`. + > - You will need to wait if there are already tasks in the task queue for the machine. + > - Your job will be randomly assigned to a machine that meets the requirements if the testbox name is not end with '-axx'. + +## Specify the OS + + About supportted `os`, `os_arch`, `os_version`, reference to [os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/ma… -- 2.23.0

1 0

[PATCH lkp-tests] lib/log.sh: add square bracket
by Liu Yinsi 26 Jan '21

26 Jan '21

before: 2021-01-26 14:53:36 ERROR -- cifs module is not loaded in after: 2021-01-26 15:00:27 [ERROR] -- cifs module is not loaded in [why] use square bracket make log more obvious. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/log.sh b/lib/log.sh index 5acf61825..207795ece 100644 --- a/lib/log.sh +++ b/lib/log.sh @@ -2,25 +2,25 @@ log() { - echo "$(date +'%F %T') INFO -- $*" + echo "$(date +'%F %T') [INFO] -- $*" } log_info() { - echo "$(date +'%F %T') INFO -- $*" + echo "$(date +'%F %T') [INFO] -- $*" } log_debug() { - echo "$(date +'%F %T') DEBUG -- $*" + echo "$(date +'%F %T') [DEBUG] -- $*" } log_warn() { - echo "$(date +'%F %T') WARN -- $*" >&2 + echo "$(date +'%F %T') [WARN] -- $*" >&2 } log_error() { - echo "$(date +'%F %T') ERROR -- $*" >&2 + echo "$(date +'%F %T') [ERROR] -- $*" >&2 } -- 2.23.0

1 0

[PATCH lkp-tests] feat: support a new field for local disk: rootfs_disk
by Yu Chuan 26 Jan '21

26 Jan '21

[Why] We now need auto install iso to the specific disk of testbox, so in order to avoid conflict with rootfs_partition, we support a new field for local disk: rootfs_disk. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- filters/run_on_local_disk | 8 ++++---- lib/bootstrap.sh | 23 +++++++++++++++++++++++ lkp-exec/qemu | 7 ++++++- 3 files changed, 33 insertions(+), 5 deletions(-) diff --git a/filters/run_on_local_disk b/filters/run_on_local_disk index 389d1ca17fdc..2092bd2b6028 100755 --- a/filters/run_on_local_disk +++ b/filters/run_on_local_disk @@ -1,11 +1,11 @@ #!/bin/bash -# check if testbox has rootfs_partition to run the job +# check if testbox has rootfs_partition or rootfs_disk to run the job if [[ "$run_on_local_disk" == "yes" || "$run_on_local_disk" == "true" ]]; then - if [[ -z "$rootfs_partition" ]]; then - echo "rm $job_file due to no rootfs_partition" - echo "run_on_local_disk needs rootfs_partition" + if [[ -z "$rootfs_partition" && -z "$rootfs_disk" ]]; then + echo "rm $job_file due to no rootfs_partition or rootfs_disk" + echo "run_on_local_disk needs rootfs_partition or rootfs_disk" exit 1 fi fi diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd48b7..6c0055bd19f5 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -490,6 +490,12 @@ wait_load_disk() ls $rootfs_partition >/dev/null 2>&1 && return # eg: LABEL=LKP-ROOTFS blkid | grep -q ${rootfs_partition#*=} && return + + # eg: /dev/disk/by-id/ata-WDC_WD1002FAEX-00Z3A0_WD-WCATRC577623 + ls $rootfs_disk >/dev/null 2>&1 && return + # eg: LABEL=LKP-ROOTFS + blkid | grep -q ${rootfs_disk#*=} && return + sleep 1 done @@ -515,6 +521,23 @@ mount_rootfs() mkdir -p $ROOTFS_DIR/tmp CACHE_DIR=$ROOTFS_DIR/tmp cleanup_pkg_cache $CACHE_DIR + elif [ -n "$rootfs_disk" ]; then + # wait for the machine to load the disk + wait_load_disk || { + # skipping following test if disk can't be load + echo "can't load the disk $rootfs_disk, skip testing..." + set_job_state 'load_disk_fail' + return 1 + } + ROOTFS_DIR=/opt/rootfs + mkdir -p $ROOTFS_DIR + mount $rootfs_disk $ROOTFS_DIR || { + mkfs.btrfs -f $rootfs_disk + mount $rootfs_disk $ROOTFS_DIR + } + mkdir -p $ROOTFS_DIR/tmp + CACHE_DIR=$ROOTFS_DIR/tmp + cleanup_pkg_cache $CACHE_DIR else CACHE_DIR=/tmp/cache mkdir -p $CACHE_DIR diff --git a/lkp-exec/qemu b/lkp-exec/qemu index 9e4b576cf50f..1cbd316f6146 100755 --- a/lkp-exec/qemu +++ b/lkp-exec/qemu @@ -91,14 +91,16 @@ replace_script_partition_val() local nr_hdd_vdisk="$(echo $hdd_partitions | wc -w)" local nr_ssd_vdisk="$(echo $ssd_partitions | wc -w)" local nr_swap_vdisk="$(echo $swap_partitions | wc -w)" + local nr_rootfs_vdisk="$(echo $rootfs_partition | wc -w)" + [ -z "$rootfs_partition" ] && nr_rootfs_vdisk="$(echo $rootfs_disk | wc -w)" [[ $nr_hdd_partitions ]] || nr_hdd_partitions=$nr_hdd_vdisk [[ $nr_ssd_partitions ]] || nr_ssd_partitions=$nr_ssd_vdisk VDISK_NUM=$((nr_hdd_partitions+nr_ssd_partitions+nr_swap_vdisk+nr_rootfs_vdisk)) - [[ "$hdd_partitions$ssd_partitions$swap_partitions$rootfs_partition" =~ '/dev/vda' ]] && return + [[ "$hdd_partitions$ssd_partitions$swap_partitions$rootfs_partition$rootfs_disk" =~ '/dev/vda' ]] && return if ((VDISK_NUM)); then local index=0 @@ -115,6 +117,9 @@ replace_script_partition_val() -e "s%export swap_partitions=.*%export swap_partitions='${vdisk_swap_val}'%" \ -e "s%export rootfs_partition=.*%export rootfs_partition='${vdisk_rootfs_val}'%" \ $job_script + + [ -z "$rootfs_partition" ] && + sed -i -e "s%export rootfs_disk=.*%export rootfs_disk='${vdisk_rootfs_val}'%" $job_script fi } -- 2.23.0

3 5

[PATCH compass-ci] update apply account email address
by Li Yuanchao 26 Jan '21

26 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI �� git repos�� **��** -1. �� compass-ci-reply(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� +1. �� compass-ci-robot(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� 2. ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9C%AC%E5%9C%B0%E5%AE%89%E8%A3%85compass-ci%E5%AE%A2%E6%88%B7%E7%AB%AF.md)�� 3. �� sshd ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%99%BB%E5%BD%95%E6%B5%8B%E8%AF%95%E6%9C%BA%E8%B0%83%E6%B5%8B%E4%BB%BB%E5%8A%A1.md)�� -- 2.23.0

1 0

[PATCH v13 compass-ci] lib/es_jobs.rb: optimize es-jobs output
by Lu Kaiyi 26 Jan '21

26 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archlinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archlinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.summary.any_error=1: 40 kvcount.summary.any_stderr=1: 44 kvcount.summary.any_warning=1: 23 kvcount.summary.success=1: 1 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 ... raw.id.[job_state=failed]: ["crystal.122405","crystal.122406","crystal.122331","crystal.122365","crystal.122404","crystal.122438","crystal.122607","crystal.122370","crystal.122375","crystal.122410","crystal.122900","crystal.122906","crystal.122271","crystal.122451","crystal.122835","crystal.122834","crystal.122232","crystal.122403","crystal.122453","crystal.122886"] ... raw.id.[summary.any_error=1]: ["crystal.122436","crystal.122476","crystal.122803","crystal.122883","crystal.122910","crystal.122438","crystal.122452","crystal.122895","crystal.122446","crystal.122685","crystal.122687","crystal.122833","crystal.122451","crystal.122690","crystal.122719","crystal.122882","crystal.122890","crystal.122453","crystal.122443","crystal.122435","crystal.122432","crystal.122688","crystal.122689","crystal.122886"] ... sum.stats.last_state.test.iperf.exit_code.127: 12 sum.stats.stderr./lkp/lkp/src/lib/env.sh:#:set_perf_path: 12 sum.stats.last_state.is_incomplete_run: 12 sum.stats.kmsg.timestamp:last: 45 sum.stats.last_state.exit_fail: 20 sum.stats.stderr.has_stderr: 44 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0,null,null,null,null,null,null,null,null,524288.0,524288.0,null,null,null,null,null,516608.0,null,1048576.0,null,null,null,null,null,null,null,524288.0,null,null,null,null,2104832.0,1572864.0,null,null,2097152.0,null,null,null,null,null,null,null,null,null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... max.stats.sched_debug.cfs_rq:/.load.stddev: 2104832.0 max.stats.softirqs.CPU1.NET_RX: 6 max.stats.slabinfo.kmalloc-512.active_objs: 1616 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 213 ++++++++++++++++++++++++++++--------------------- 1 file changed, 123 insertions(+), 90 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..c35b182 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -3,137 +3,170 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo summary.success + summary.any_fail summary.any_error summary.any_stderr summary.any_warning].freeze + +require 'json' require "#{LKP_SRC}/lib/stats" require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @es_query = es_query - @es = ESQuery.new(ES_HOST, ES_PORT) + @jobs = query_jobs_from_es(es_query) @refine = my_refine @fields = fields @stats_filter = stats_filter - @stats_filter_result = {} @refine_jobs = [] - @jobs = {} - @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' - } - set_defaults - deal_jobs + set_jobs_summary end - def set_defaults - query_result = @es.multi_field_query(@es_query) - query_result['hits']['hits'].each do |job| - @jobs[job['_id']] = job['_source'] - end - - @stats = { - 'stats.count' => Hash.new(0), - 'stats.sum' => Hash.new(0), - 'stats.avg' => Hash.new(0) - } - @result = {} - @fields.each do |field| - @result[field] = [] - end + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs end - def add_result_fields(job, level) - return unless @refine.include?(level) || @refine.include?(-1) - - @refine_jobs << job['id'] - @fields.each do |field| - value = job[field] - if value - value = job['id'] + '.' + value if field == 'job_state' - @result[field] << value + def set_job_summary(stats, job) + summary_result = '' + stats.each_key do |stat| + # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, + # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, + # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, + if stat.match(/warning/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "stderr.linux-perf": 1, + # "stderr.error:target_not_found:ruby-dev": 1, + # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, + if stat.match(/stderr\./i) + job['summary.any_stderr'] = 1 + summary_result = 'stderr' + end + # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, + # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, + # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + # "last_state.test.iperf.exit_code.127": 1, + # "last_state.test.cci-makepkg.exit_code.1": 1, + if stat.match(/error|\.exit_code\./i) + job['summary.any_error'] = 1 + summary_result = 'error' + end + if stat.match(/\.fail$/i) + job['summary.any_fail'] = 1 + summary_result = 'fail' end - - next unless job['stats'] - - @result[field] << job['stats'][field] if job['stats'][field] end - end - - def deal_jobs - stats_count = Hash.new(0) - stats_jobs = {} + return unless summary_result.empty? - @jobs.each do |job_id, job| - level = deal_stats(job) - add_result_fields(job, level) + job['summary.success'] = 1 + end - stat_key = @stats_level[level] - stat_jobs_key = stat_key + '_jobs' + # set jobs summary fields information in place + def set_jobs_summary + @jobs.each do |job| + stats = job['stats'] + next unless stats - stats_count[stat_key] += 1 - stats_jobs[stat_jobs_key] ||= [] - stats_jobs[stat_jobs_key] << job_id + set_job_summary(stats, job) end - - @stats['stats.count'].merge!(stats_count) - @stats['stats.count'].merge!(stats_jobs) end - def deal_stats(job, level = 0) - return 1 unless job['stats'] + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats - job['stats'].each do |key, value| - match_stats_filter(key, value, job['id']) - calculate_stat(key, value) - level = get_stat_level(key, level) + metrics.concat(stats.keys) end - return level + metrics.uniq! end - def match_stats_filter(key, value, job_id) - @stats_filter.each do |filter| - next unless key.include?(filter) + def initialize_result_hash(metrics) + result = { + 'kvcount' => {}, + 'raw.id' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} + } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end - key = job_id + '.' + key - @stats_filter_result[key] = value + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - break + stats.each do |key, value| + result['raw.stats'][key] << value end end - def calculate_stat(key, value) - if function_stat?(key) - return unless @fields.include?('stats.sum') + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats['stats.sum'][key] += value - else - return unless @fields.include?('stats.avg') + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] + result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] + end + end - @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end end end - def get_stat_level(stat, level) - return level if level >= 3 - return 3 if stat.match(/error|fail/i) - return 2 if stat.match(/warn/i) + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end - return 0 + stats_count(result) + result end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def output_yaml(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + if value.is_a? Hash + output_yaml(prefix_key, value) + else + puts "#{prefix_key}: #{value.to_json}" + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state(@jobs) + @result['kvcount'] = @result['kvcount'].sort.to_h + @result['raw.id'] = @result['raw.id'].sort.to_h + output_yaml('', @result) end end -- 2.23.0

2 2

[PATCH compass-ci] lib/git: use work_dir replace repo for GitCommit class
by Cao Xueliang 26 Jan '21

26 Jan '21

If we use "git -C /srv/git/#{repo}.git", we need mount "/srv/git", but sometimes we have clone the git repo, so we don't need mount "/srv/git". Now, we just need to pass the work_dir to the GitCommit class. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git.rb | 4 ++-- lib/git_bisect.rb | 2 +- lib/mail_bisect_result.rb | 8 +++++++- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/lib/git.rb b/lib/git.rb index db97c47..9751dcf 100644 --- a/lib/git.rb +++ b/lib/git.rb @@ -2,8 +2,8 @@ # wrap common git commands class GitCommit - def initialize(repo, commit) - @git_prefix = "git -C /srv/git/#{repo}.git" + def initialize(work_dir, commit) + @git_prefix = "git -C #{work_dir}" @commit = commit end diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index e577afd..5d13074 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -100,10 +100,10 @@ class GitBisect bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_error_id(@error_id) all_errors = Utils.obt_errors(@work_dir, first_bad_commit) first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) - FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, + 'work_dir' => @work_dir, 'bisect_error' => bisect_error, 'all_errors' => all_errors, 'pkgbuild_repo' => @pkgbuild_repo, diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 209a28d..637c008 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -12,12 +12,13 @@ require_relative 'assign_account_client' class MailBisectResult def initialize(bisect_info) @repo = bisect_info['repo'] + @work_dir = bisect_info['work_dir'] @commit_id = bisect_info['commit'] @all_errors = bisect_info['all_errors'] @bisect_error = bisect_info['bisect_error'] @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] - @git_commit = GitCommit.new(@repo, @commit_id) + @git_commit = GitCommit.new(@work_dir, @commit_id) @to = @git_commit.author_email # now send mail to review @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' @@ -26,6 +27,7 @@ class MailBisectResult def create_send_email send_report_mail(compose_mail) send_account_mail + rm_work_dir end def compose_mail @@ -80,4 +82,8 @@ class MailBisectResult apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end + + def rm_work_dir + FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) + end end -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs: fix error when search nothing
by Wu Zhende 26 Jan '21

26 Jan '21

[Before] es-jobs suite=test Traceback (most recent call last): 3: from /home/wuzhende/compass-ci/sbin/es-jobs:55:in `<main>' 2: from /home/wuzhende/compass-ci/lib/es_jobs.rb:167:in `output' 1: from /home/wuzhende/compass-ci/lib/es_jobs.rb:137:in `query_jobs_state' /home/wuzhende/compass-ci/lib/es_jobs.rb:99:in `initialize_result_hash': undefined method `each' for nil:NilClass (NoMethodError) [After] es-jobs suite=test No query result is found: {"suite"=>"test"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index c35b182..eb3f70d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -13,7 +13,8 @@ require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @jobs = query_jobs_from_es(es_query) + @es_query = es_query + @jobs = query_jobs_from_es @refine = my_refine @fields = fields @stats_filter = stats_filter @@ -21,9 +22,9 @@ class ESJobs set_jobs_summary end - def query_jobs_from_es(items) + def query_jobs_from_es es = ESQuery.new(ES_HOST, ES_PORT) - result = es.multi_field_query items + result = es.multi_field_query @es_query jobs = result['hits']['hits'] jobs.map! { |job| job['_source'] } return jobs @@ -164,6 +165,10 @@ class ESJobs end def output + if @jobs.empty? + puts "No query result is found: #{@es_query}" + return + end @result = query_jobs_state(@jobs) @result['kvcount'] = @result['kvcount'].sort.to_h @result['raw.id'] = @result['raw.id'].sort.to_h -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/apply-account: the repo should be in our test coverage
by Wu Fengguang 26 Jan '21

26 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index b5430e7..3e253d0 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -27,8 +27,9 @@ Email template: - please replace the 3 parts in {{ }} with your information. - the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. -- my oss commit: a git commit URL that has your contribution. It should be in a - well known git repo that's registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos) +- my oss commit: a git commit URL that has your name and email. + The git repo should be in our test coverage and registered in + [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). ## step 2: receive an email -- 2.23.0

1 0

[PATCH v6 compass-ci] lib/sched_client.rb: add some common methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler [how] each method corresponds to a interface, so ensuring the independence of them. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..93f9ccd 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,37 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type, parameter, value) + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + def register_mac2host(hostname, mac) + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{hostname}&mac=#{mac}", {} + ) + end + + def register_host2queues(hostname, queues) + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{hostname}&queues=#{queues}", {} + ) + end + + def delete_mac2host(mac) + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{mac}", {} + ) + end + + def delete_host2queues(hostname) + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{hostname}", {} + ) + end end -- 2.23.0

1 0

[PATCH lab-z9] add hosts-file of taishan200-2280-2s64p-256g--a5 which include NMVe SSD
by Zhang Dewan 26 Jan '21

26 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s64p-256g--a5 | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a5 diff --git a/hosts/taishan200-2280-2s64p-256g--a5 b/hosts/taishan200-2280-2s64p-256g--a5 new file mode 100644 index 0000000..fc9a594 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a5 @@ -0,0 +1,26 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981306c5 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43957 + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000328 +mac_addr: + - 84:46:fe:26:d3:47 + - 84:46:fe:26:d3:48 + - 84:46:fe:26:d3:49 + - 84:46:fe:26:d3:4a + - ac:b3:b5:1e:b4:ab + - ac:b3:b5:1e:b4:ac + - ac:b3:b5:1e:b4:ad + - ac:b3:b5:1e:b4:ae + - ac:b3:b5:1e:ba:9b + - ac:b3:b5:1e:ba:9c + - ac:b3:b5:1e:ba:9d + - ac:b3:b5:1e:ba:9e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.5 -- 2.23.0

2 2

[PATCH v5 compass-ci] lib/sched_client.rb: add some common methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..a81ffbb 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,41 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type, parameter, value) + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + def register_mac2host(hostname, mac) + @hostname = hostname + @mac = mac + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{@hostname}&mac=#{@mac}", {} + ) + end + + def register_host2queues(hostname, queues) + @hostname = hostname + @queues = queues + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{@hostname}&queues=#{@queues}", {} + ) + end + + def delete_mac2host + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{@mac}", {} + ) + end + + def delete_host2queues + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{@hostname}", {} + ) + end end -- 2.23.0

2 2

[PATCH v2 compass-ci] container: expose/report error when check modules exist
by Liu Yinsi 26 Jan '21

26 Jan '21

[why] sparrow/2-network/nfs: modprobe nfs modprobe nfsd sparrow/2-network/cifs: modprobe cifs when locally deploy compass-ci, the above codes have modprobe nfs, nfsd, cifs, no need to do again when execute sparrow/4-docker/buildall(deploy containers), exit and report error message to function callers is better. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 3 ++- container/os-cifs/start | 3 ++- container/os-nfs/start | 6 ++++-- container/result-cifs/start | 3 ++- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 77558a1..c70b318 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm initrd-cifs diff --git a/container/os-cifs/start b/container/os-cifs/start index 5e08238..6df167f 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm os-cifs diff --git a/container/os-nfs/start b/container/os-nfs/start index 8b8ff5f..3b0ebb5 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -5,10 +5,12 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^nfs\s" || { - sudo modprobe nfs + echo "[ERROR] nfs module is not loaded in the Docker host's kernel (try: sudo modprobe nfs)" + exit } lsmod | grep -q "^nfsd\s" || { - sudo modprobe nfsd + echo "[ERROR] nfsd module is not loaded in the Docker host's kernel (try: sudo modprobe nfsd)" + exit } # Several nfs servers running will cause conflict, so rm an old before running a new nfs server docker. diff --git a/container/result-cifs/start b/container/result-cifs/start index 2d6acba..3992cdd 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm result-cifs -- 2.23.0

2 4

[PATCH compass-ci] sparrow/3-code: set params
by Liu Yinsi 26 Jan '21

26 Jan '21

before: deploy compass-ci client and server in one machine. after: not only deploy compass-ci client and server in one machine, but also on different servers. [why] for user locally deploy compass-ci, sometimes need to deploy client in other machine, so add $1. it's interrelated with the follow patch: [PATCH lkp-tests] setup/simplify-ci: delete default value Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 2ba9662..00a2e0f 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -4,6 +4,9 @@ server_name=$(hostname | cut -f1 -d.) server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') + +[ -n "$1" ] && server_ip=$1 + : ${sched_host:=$server_ip} : ${sched_port:=3000} : ${os_http_host:=$server_ip} -- 2.23.0

1 0

[PATCH v2 lkp-tests] lkp-bootstrap.service: fix killmode error
by Wang Chenglong 26 Jan '21

26 Jan '21

[Error] Welcome to openEuler 20.09! /usr/lib/systemd/system/lkp-bootstrap.service:11: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the ser vice. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. [How] config the lkp-bootstrap.service: KillMode=control-group Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service b/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service index 842c4aef8..a9eb5152a 100644 --- a/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service +++ b/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service @@ -8,7 +8,7 @@ Type=forking Environment=HOME=/root ExecStart=/etc/init.d/lkp-bootstrap IgnoreSIGPIPE=no -KillMode=none +KillMode=control-group TasksMax=infinity [Install] -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/apply-account: add introduction
by Wu Fengguang 26 Jan '21

26 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index a0b2c82..b5430e7 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -1,5 +1,15 @@ # apply account +You'll need an account to submit jobs. The account is mainly for tracking your resource consumptions. +We have limited pool of testboxes, so there have to be some usage control. + +We offer free accounts to OSS (Open Source Software) contributors. +You can prove yourself as OSS contributor by providing "my oss commit" in the below email. + +We offer free accounts to our collaborators too. If you are one of our +collaborators, you should know the collaboration channel to apply account. +In this case you just need provide name, email, ssh pubkey and purpose. + ## step 1: send email to apply account Email template: @@ -16,8 +26,9 @@ Email template: --- - please replace the 3 parts in {{ }} with your information. -- the email name should be in english, as: "David Rientjes" in the template. -- my oss commit: a git commit URL that has your contribution. +- the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. +- my oss commit: a git commit URL that has your contribution. It should be in a + well known git repo that's registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos) ## step 2: receive an email @@ -29,7 +40,7 @@ Which contains account information for you: ## step 3: one-time setup -**Follow the email** to setup your local environment: +**Follow instructions in the email** to setup your local environment: git clone https://gitee.com/wu_fengguang/lkp-tests.git setup ~/.config/compass-ci/defaults/account.yaml @@ -38,10 +49,3 @@ Which contains account information for you: Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -## notice: - -In case that you received a failed email with message as below: - - The repo url for your commit is not in our upstream-repo list. - -Try to [register repo to upstream-repos](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/m… -- 2.23.0

1 0

[PATCH compass-ci] uniform coding style
by Li Ping 26 Jan '21

26 Jan '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 12c1f3e..7760c93 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -204,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs = nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix -- 2.23.0

1 0

[PATCH v3 compass-ci 5/5] upload-libvirt-xml/nginx.conf: nginx.conf in upload-libvirt-xml container
by cuiyili 26 Jan '21

26 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/nginx.conf | 43 +++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 container/upload-libvirt-xml/nginx.conf diff --git a/container/upload-libvirt-xml/nginx.conf b/container/upload-libvirt-xml/nginx.conf new file mode 100644 index 0000000..aef8fb8 --- /dev/null +++ b/container/upload-libvirt-xml/nginx.conf @@ -0,0 +1,43 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user lkp; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + server { + listen 3081; + server_name "upload-libvirt-xml"; + server_tokens off; + client_max_body_size 50m; + access_log /tmp/access.log; + error_log /tmp/error.log; + + location / { + allow all; + root /srv/cci/libvirt-xml/; + + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + content_by_lua_block { + ngx.req.read_body() + local filename = string.gsub(ngx.var.request_uri, "/", "") + if string.match(filename, "([^/]+).xml$") then + add_path = string.match(filename, "([^/]+).xml$") + local data = ngx.req.get_body_data() + local file = io.open("/srv/cci/libvirt-xml/"..filename, "w") + file:write(data) + file:close() + file_md5 = io.popen("cd /srv/cci/libvirt-xml/ && file_md5=$(md5sum "..filename.." | awk '{print $1}') && mkdir -p "..add_path.." && mv "..filename.." "..add_path.."/${file_md5}.xml && echo ${file_md5}"):read() + ngx.say(add_path.."/"..tostring(file_md5)..".xml") + end + } + } + + } +} -- 2.23.0

3 2

[PATCH v2 compass-ci] container: fix maintainer email address format
by Liu Yinsi 26 Jan '21

26 Jan '21

1. maintainer email address format use: xxx <xxx(a)xxx.com> 2. Update maintainers, shellcheck for Bai Jing <799286817(a)qq.com> web-backend-nginx and web-backend for Lu Weitao <luweitaobe(a)163.com> FYI, Baijing and Weitao, please help to maintain the containers. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/assist-result/Dockerfile | 2 +- container/assistant/Dockerfile | 2 +- container/send-internet-mail/Dockerfile | 2 +- container/send-mail/Dockerfile | 2 +- container/shellcheck/Dockerfile | 2 +- container/web-backend-nginx/Dockerfile | 2 +- container/web-backend/Dockerfile | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 60a2fbf..ce124b9 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -3,7 +3,7 @@ FROM alpine:3.11 -MAINTAINER caoxl78320(a)163.com +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile index f1ad7a2..04d2da0 100644 --- a/container/assistant/Dockerfile +++ b/container/assistant/Dockerfile @@ -3,7 +3,7 @@ FROM debian -MAINTAINER caoxl78320(a)163.com +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ENV DEBIAN_FRONTEND noninteractive diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index 03e3327..29f1bc6 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER shdluan(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index d151bda..b334b07 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER shdluan(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ diff --git a/container/shellcheck/Dockerfile b/container/shellcheck/Dockerfile index b78beb7..0bfe7c0 100644 --- a/container/shellcheck/Dockerfile +++ b/container/shellcheck/Dockerfile @@ -3,7 +3,7 @@ FROM debian -MAINTAINER 18509160991(a)163.com +MAINTAINER Bai Jing <799286817(a)qq.com> ENV DEBIAN_FRONTEND noninteractive diff --git a/container/web-backend-nginx/Dockerfile b/container/web-backend-nginx/Dockerfile index a3882ac..bbf7af2 100644 --- a/container/web-backend-nginx/Dockerfile +++ b/container/web-backend-nginx/Dockerfile @@ -3,6 +3,6 @@ FROM nginx:alpine -MAINTAINER Zhang Yuhang <o0OiIl(a)hotmail.com> +MAINTAINER Lu Weitao <luweitaobe(a)163.com> COPY nginx.conf /etc/nginx/ diff --git a/container/web-backend/Dockerfile b/container/web-backend/Dockerfile index 2ac5148..a26c167 100644 --- a/container/web-backend/Dockerfile +++ b/container/web-backend/Dockerfile @@ -2,7 +2,7 @@ FROM debian -MAINTAINER Zhang Yuhang <o0OiIl(a)hotmail.com> +MAINTAINER Lu Weitao <luweitaobe(a)163.com> ENV DEBIAN_FRONTEND noninteractive -- 2.23.0

1 0

[PATCH compass-ci] container: Fix maintainer email address format
by Liu Yinsi 26 Jan '21

26 Jan '21

1. maintainer email address format use: xxx <xxx(a)xxx.com> 2. Update maintainers, shellcheck for Bai Jing <799286817(a)qq.com> web-backend-nginx and web-backend for Lu Weitao <luweitaobe(a)163.com> FYI, Baijing and Weitao, please help to maintain the containers. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/assist-result/Dockerfile | 2 +- container/assistant/Dockerfile | 2 +- container/send-internet-mail/Dockerfile | 2 +- container/send-mail/Dockerfile | 2 +- container/shellcheck/Dockerfile | 2 +- container/web-backend-nginx/Dockerfile | 2 +- container/web-backend/Dockerfile | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 60a2fbf..ce124b9 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -3,7 +3,7 @@ FROM alpine:3.11 -MAINTAINER caoxl78320(a)163.com +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile index f1ad7a2..04d2da0 100644 --- a/container/assistant/Dockerfile +++ b/container/assistant/Dockerfile @@ -3,7 +3,7 @@ FROM debian -MAINTAINER caoxl78320(a)163.com +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ENV DEBIAN_FRONTEND noninteractive diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index 03e3327..29f1bc6 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER shdluan(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index d151bda..b334b07 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER shdluan(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ diff --git a/container/shellcheck/Dockerfile b/container/shellcheck/Dockerfile index b78beb7..0bfe7c0 100644 --- a/container/shellcheck/Dockerfile +++ b/container/shellcheck/Dockerfile @@ -3,7 +3,7 @@ FROM debian -MAINTAINER 18509160991(a)163.com +MAINTAINER Bai Jing <799286817(a)qq.com> ENV DEBIAN_FRONTEND noninteractive diff --git a/container/web-backend-nginx/Dockerfile b/container/web-backend-nginx/Dockerfile index a3882ac..bbf7af2 100644 --- a/container/web-backend-nginx/Dockerfile +++ b/container/web-backend-nginx/Dockerfile @@ -3,6 +3,6 @@ FROM nginx:alpine -MAINTAINER Zhang Yuhang <o0OiIl(a)hotmail.com> +MAINTAINER Lu Weitao <luweitaobe(a)163.com> COPY nginx.conf /etc/nginx/ diff --git a/container/web-backend/Dockerfile b/container/web-backend/Dockerfile index 2ac5148..a26c167 100644 --- a/container/web-backend/Dockerfile +++ b/container/web-backend/Dockerfile @@ -2,7 +2,7 @@ FROM debian -MAINTAINER Zhang Yuhang <o0OiIl(a)hotmail.com> +MAINTAINER Lu Weitao <luweitaobe(a)163.com> ENV DEBIAN_FRONTEND noninteractive -- 2.23.0

1 1

[PATCH v3 compass-ci 1/5] upload-libvirt-xml/build: add upload-libvirt-xml docker image
by cuiyili 26 Jan '21

26 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/build | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100755 container/upload-libvirt-xml/build diff --git a/container/upload-libvirt-xml/build b/container/upload-libvirt-xml/build new file mode 100755 index 0000000..d39838b --- /dev/null +++ b/container/upload-libvirt-xml/build @@ -0,0 +1,10 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# The libvirt xml upload has nothing to do with the job. It renames the uploaded file and does not currently have permission to upload. +# it is a transitional service. Two different functions use the same port service, + +docker build -t upload-libvirt-xml . -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/sched_client.rb: add some common methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..db60fa2 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,43 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type: '', parameter: '', value: '') + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + # some provider only need register host2queues + # like boot.container + def register_mac2host(hostname, mac) + @hostname = hostname + @mac = mac + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{@hostname}&mac=#{@mac}", {} + ) + end + + def register_host2queues(hostname, queues) + @hostname = hostname + @queues = queues + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{@hostname}&queues=#{@queues}", {} + ) + end + + def delete_mac2host + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{@mac}", {} + ) + end + + def delete_host2queues + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{@hostname}", {} + ) + end end -- 2.23.0

3 4

[PATCH v4 compass-ci 2/2] container/lifecycle: add a new docker service
by Wu Zhende 26 Jan '21

26 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 25 +++++++++++++++++++++++ container/lifecycle/build | 18 +++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 33 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 78 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..aa3e737 --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER Wu Zhende <wuzhende666(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install rest-client activesupport git json yaml + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..ba68a85 --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..7e67b3c --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,33 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] service/lifecycle: add a new service
by Wu Zhende 26 Jan '21

26 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..af5cd07 --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "Lifecycle Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def get_running_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..4154a27 --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find the testbox that are performing jobs + # curl http://localhost:11311/get_running_testbox?size=10&from=0 + get "/get_running_testbox" do |env| + env.lifecycle.get_running_testbox.to_json + end +end -- 2.23.0

1 0

[PATCH v4 compass-ci] lib/sched_client.rb: add some common methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..b317790 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,43 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type, parameter, value) + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + # some provider only need register host2queues + # like boot.container + def register_mac2host(hostname, mac) + @hostname = hostname + @mac = mac + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{@hostname}&mac=#{@mac}", {} + ) + end + + def register_host2queues(hostname, queues) + @hostname = hostname + @queues = queues + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{@hostname}&queues=#{@queues}", {} + ) + end + + def delete_mac2host + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{@mac}", {} + ) + end + + def delete_host2queues + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{@hostname}", {} + ) + end end -- 2.23.0

2 2

[PATCH v4 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 26 Jan '21

26 Jan '21

add limit for borrowing machine: runtime <= 10 days only jobs that runtime value is less than or equal to 10 days are allowed for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..2c3007c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,18 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + + # case runtime value is more than 10 days, job will fail. + raise error_msg if hash["runtime"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0