- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci] save hostname and queues relation in redis
by Xiao Shenwei 02 Nov '20

02 Nov '20

[why] the test machine has the ability to know which queues should be requested [how] for VM HW, we will register mac2host and host2queues. then they can find queues based on mac. API: boot.ipxe/mac/${mac} for docker, we will register host2queues, then container can find queues based on hostname. API: boot.container/hostname/${hostname Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/sched.cr | 8 ++++++++ src/scheduler/scheduler.cr | 24 ++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index b2ef6ce..80cf0b0 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -38,6 +38,14 @@ class Sched @redis.hash_del("sched/mac2host", normalize_mac(mac)) end + def set_host2queues(hostname : String, queues : String) + @redis.hash_set("sched/host2queues", hostname, queues) + end + + def del_host2queues(hostname : String) + @redis.hash_del("sched/host2queues", hostname) + end + # return: # Hash(String, Hash(String, String)) def get_cluster_state(cluster_id) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 371e4ed..a946ea0 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -113,6 +113,18 @@ module Scheduler end end + # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" + put "/set_host2queues" do |env| + env.response.headers["Connection"] = "close" + if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) + sched.set_host2queues(client_host, client_queues) + + "Done" + else + "No yet" + end + end + # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| env.response.headers["Connection"] = "close" @@ -125,6 +137,18 @@ module Scheduler end end + # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" + put "/del_host2queues" do |env| + env.response.headers["Connection"] = "close" + if client_host = env.params.query["host"]? + sched.del_host2queues(client_host) + + "Done" + else + "No yet!" + end + end + # client(runner) report job's status # /~lkp/cgi-bin/lkp-jobfile-append-var # ?job_file=/lkp/scheduled/job.yaml&job_state=running&job_id=10 -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 02 Nov '20

02 Nov '20

auto group jobs with user-defined template to support template compare mode background: before compare, create job-groups for all jobs that query from ES with user's filter then, create matrices-groups according job-groups finally, compare each group matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 121 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..3c6de7a 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,122 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } +# + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + next unless job['stats'] + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + new_job = get_new_job_by_metrics(job, metrics) + next if new_job.empty? + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + if job.key?(param) + group_key_list << job[param] + next + end + job.each_value do |v| + if v.is_a?(Hash) && v.key?(param) + group_key_list << v[param] + break + end + end + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job_by_metrics(job, metrics) + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

3 7

[PATCH v3 compass-ci] LICENSES: add THIRD PARTY OPEN SOURCE SOFTWARE NOTICE
by Lu Kaiyi 02 Nov '20

02 Nov '20

add THIRD PARTY OPEN SOURCE SOFTWARE NOTICE for references. the directory structure of LICENSES as below: . ├── CCBY-4.0 ├── coreutils │ ├── coreutils │ ├── files │ └── GPL-3.0 -> ../GPL-3.0 ├── docker-sshd │ ├── docker-sshd │ ├── files │ └── MIT -> ../MIT ├── dracut │ ├── dracut │ ├── files │ └── GPL-2.0 -> ../GPL-2.0 ├── GPL-2.0 ├── GPL-3.0 ├── kemal │ ├── files │ ├── kemal │ └── MIT -> ../MIT ├── lkp-tests │ ├── files │ ├── GPL-2.0 -> ../GPL-2.0 │ └── lkp-tests ├── MIT ├── mritd.github.io │ ├── files │ ├── MIT -> ../MIT │ └── mritd.github.io ├── MulanPSL-2.0 └── Scout ├── files ├── Scout └── MIT -> ../MIT Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- LICENSES/Scout/MIT | 1 + LICENSES/Scout/Scout | 17 +++++++++++++++++ LICENSES/Scout/files | 1 + LICENSES/coreutils/GPL-3.0 | 1 + LICENSES/coreutils/coreutils | 17 +++++++++++++++++ LICENSES/coreutils/files | 1 + LICENSES/docker-sshd/MIT | 1 + LICENSES/docker-sshd/docker-sshd | 17 +++++++++++++++++ LICENSES/docker-sshd/files | 1 + LICENSES/dracut/GPL-2.0 | 1 + LICENSES/dracut/dracut | 18 ++++++++++++++++++ LICENSES/dracut/files | 1 + LICENSES/kemal/MIT | 1 + LICENSES/kemal/files | 1 + LICENSES/kemal/kemal | 17 +++++++++++++++++ LICENSES/lkp-tests/GPL-2.0 | 1 + LICENSES/lkp-tests/files | 8 ++++++++ LICENSES/lkp-tests/lkp-tests | 18 ++++++++++++++++++ LICENSES/mritd.github.io/MIT | 1 + LICENSES/mritd.github.io/files | 1 + LICENSES/mritd.github.io/mritd.github.io | 17 +++++++++++++++++ 21 files changed, 142 insertions(+) create mode 120000 LICENSES/Scout/MIT create mode 100644 LICENSES/Scout/Scout create mode 100644 LICENSES/Scout/files create mode 120000 LICENSES/coreutils/GPL-3.0 create mode 100644 LICENSES/coreutils/coreutils create mode 100644 LICENSES/coreutils/files create mode 120000 LICENSES/docker-sshd/MIT create mode 100644 LICENSES/docker-sshd/docker-sshd create mode 100644 LICENSES/docker-sshd/files create mode 120000 LICENSES/dracut/GPL-2.0 create mode 100644 LICENSES/dracut/dracut create mode 100644 LICENSES/dracut/files create mode 120000 LICENSES/kemal/MIT create mode 100644 LICENSES/kemal/files create mode 100644 LICENSES/kemal/kemal create mode 120000 LICENSES/lkp-tests/GPL-2.0 create mode 100644 LICENSES/lkp-tests/files create mode 100644 LICENSES/lkp-tests/lkp-tests create mode 120000 LICENSES/mritd.github.io/MIT create mode 100644 LICENSES/mritd.github.io/files create mode 100644 LICENSES/mritd.github.io/mritd.github.io diff --git a/LICENSES/Scout/MIT b/LICENSES/Scout/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/Scout/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/Scout/Scout b/LICENSES/Scout/Scout new file mode 100644 index 0000000..6844406 --- /dev/null +++ b/LICENSES/Scout/Scout @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: Scout db170ce4ff160c4a2c30483e9410df1773cf572c + +Copyright notice: Copyright(c) 2010 Menno van Slooten, http://mennovanslooten.nl/ +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/Scout/files b/LICENSES/Scout/files new file mode 100644 index 0000000..e120df8 --- /dev/null +++ b/LICENSES/Scout/files @@ -0,0 +1 @@ +/compass-ci/lib/themes.rb diff --git a/LICENSES/coreutils/GPL-3.0 b/LICENSES/coreutils/GPL-3.0 new file mode 120000 index 0000000..481ee24 --- /dev/null +++ b/LICENSES/coreutils/GPL-3.0 @@ -0,0 +1 @@ +../GPL-3.0 \ No newline at end of file diff --git a/LICENSES/coreutils/coreutils b/LICENSES/coreutils/coreutils new file mode 100644 index 0000000..1250a6f --- /dev/null +++ b/LICENSES/coreutils/coreutils @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: coreutils 6.9.89 + +Copyright notice: Copyright(C) 2007 Free Software Foundation, Inc. <https://fsf.org/> +License: The GPL-3.0 License +License-Text: please refer to file GPL-3.0 diff --git a/LICENSES/coreutils/files b/LICENSES/coreutils/files new file mode 100644 index 0000000..8a2b690 --- /dev/null +++ b/LICENSES/coreutils/files @@ -0,0 +1 @@ +/compass-ci/container/os-nfs/entrypoint.sh diff --git a/LICENSES/docker-sshd/MIT b/LICENSES/docker-sshd/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/docker-sshd/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/docker-sshd/docker-sshd b/LICENSES/docker-sshd/docker-sshd new file mode 100644 index 0000000..603f668 --- /dev/null +++ b/LICENSES/docker-sshd/docker-sshd @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: docker-sshd 729faee00bcf0eff5f80059cd92aa425505820f0 + +Copyright notice: Copyright(c) 2015-2020 Volt Grid Pty Ltd +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/docker-sshd/files b/LICENSES/docker-sshd/files new file mode 100644 index 0000000..018684c --- /dev/null +++ b/LICENSES/docker-sshd/files @@ -0,0 +1 @@ +/compass-ci/container/ssh-r/entry.sh diff --git a/LICENSES/dracut/GPL-2.0 b/LICENSES/dracut/GPL-2.0 new file mode 120000 index 0000000..9824569 --- /dev/null +++ b/LICENSES/dracut/GPL-2.0 @@ -0,0 +1 @@ +../GPL-2.0 \ No newline at end of file diff --git a/LICENSES/dracut/dracut b/LICENSES/dracut/dracut new file mode 100644 index 0000000..f09df60 --- /dev/null +++ b/LICENSES/dracut/dracut @@ -0,0 +1,18 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: dracut RHEL-7.1 + +Copyright notice: Copyright(C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301 USA. +License: The GPL-2.0 License +License-Text: please refer to file GPL-2.0 diff --git a/LICENSES/dracut/files b/LICENSES/dracut/files new file mode 100644 index 0000000..78235de --- /dev/null +++ b/LICENSES/dracut/files @@ -0,0 +1 @@ +/compass-ci/container/dracut-initrd/bin/cifs-lib.sh diff --git a/LICENSES/kemal/MIT b/LICENSES/kemal/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/kemal/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/kemal/files b/LICENSES/kemal/files new file mode 100644 index 0000000..71a8cab --- /dev/null +++ b/LICENSES/kemal/files @@ -0,0 +1 @@ +/compass-ci/src/spec/scheduler/boot_spec.cr diff --git a/LICENSES/kemal/kemal b/LICENSES/kemal/kemal new file mode 100644 index 0000000..e08255f --- /dev/null +++ b/LICENSES/kemal/kemal @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: kemal 0.17.0 + +Copyright notice: Copyright(c) 2016 Serdar Doğruyol +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/lkp-tests/GPL-2.0 b/LICENSES/lkp-tests/GPL-2.0 new file mode 120000 index 0000000..9824569 --- /dev/null +++ b/LICENSES/lkp-tests/GPL-2.0 @@ -0,0 +1 @@ +../GPL-2.0 \ No newline at end of file diff --git a/LICENSES/lkp-tests/files b/LICENSES/lkp-tests/files new file mode 100644 index 0000000..fde5c05 --- /dev/null +++ b/LICENSES/lkp-tests/files @@ -0,0 +1,8 @@ +/compass-ci/sbin/compare +/compass-ci/providers/multi-docker +/compass-ci/container/open-scheduler/build-depends +/compass-ci/user-client/jobs/iperf-pxe.yaml +/compass-ci/user-client/jobs/iperf-vm.yaml +/compass-ci/src/features/jobs/right_iperf.yaml +/compass-ci/lib/matrix2.rb +/compass-ci/sbin/es-find diff --git a/LICENSES/lkp-tests/lkp-tests b/LICENSES/lkp-tests/lkp-tests new file mode 100644 index 0000000..3f40d5f --- /dev/null +++ b/LICENSES/lkp-tests/lkp-tests @@ -0,0 +1,18 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: lkp-tests fcb47ccfc50c1f4da388aed9596c0acaac04a917 + +Copyright notice: Copyright(C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301 USA +License: The GPL-2.0 License +License-Text: please refer to file GPL-2.0 diff --git a/LICENSES/mritd.github.io/MIT b/LICENSES/mritd.github.io/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/mritd.github.io/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/mritd.github.io/files b/LICENSES/mritd.github.io/files new file mode 100644 index 0000000..38b67c1 --- /dev/null +++ b/LICENSES/mritd.github.io/files @@ -0,0 +1 @@ +/compass-ci/container/registry/config.yml diff --git a/LICENSES/mritd.github.io/mritd.github.io b/LICENSES/mritd.github.io/mritd.github.io new file mode 100644 index 0000000..ee22568 --- /dev/null +++ b/LICENSES/mritd.github.io/mritd.github.io @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: mritd.github.io ed4350e91e9b76078a850ede8302854d3393f493 + +Copyright notice: Copyright(c) 2013-2016 Blackrock Digital LLC. +License: The MIT License +License-Text: please refer to file MIT -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/*: fix cci job dir layout
by Wang Yong 02 Nov '20

02 Nov '20

[Why] for multi version support, cgz file for cci-makepkg need change dir layout, and change cci-depends dir layout together to easily update scheduler code path: /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/ => /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/${pkgname}/ filename: cci-makepkg ${pkgver}-${pkgrel}.cgz latest.cgz -> ${pkgver}-${pkgrel}.cgz filename: cci-depends stay original ${benchmark}.cgz ${benchmark}.cgz -> ${benchmark}_${date}.cgz E.g. output: cci-depends stay original $ tree iperf ├── iperf_20201102.cgz └── iperf.cgz -> iperf_20201102.cgz output: cci-makepkg $ tree sar ├── git-1.cgz └── latest.cgz -> git-1.cgz Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 +- tests/cci-makepkg | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 36923ea4d..efa9a025b 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -18,7 +18,7 @@ DEPS_MNT=/initrd/deps && mkdir -p "$DEPS_MNT" DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} pack_arch=$os_arch [[ "$benchmark" = "${benchmark##*.}" ]] || diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9d5082629..9551ebfe2 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -26,7 +26,7 @@ mkdir -p "$PKG_MNT" export DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} . $LKP_SRC/distro/${DISTRO} . $LKP_SRC/lib/install.sh @@ -65,8 +65,8 @@ update_shared_pkg() [ "$bm_name" = "$benchmark" ] && return # benchmark is a symlink - ln -sf "$bm_link" "$sync_dest/${benchmark}.cgz" || return - echo "update shared pkg link ${benchmark}.cgz -> $bm_link" + ln -sf latest.cgz "$sync_dest/${benchmark}.cgz" || return + echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } distro_install_depends lkp-dev @@ -86,7 +86,7 @@ date=$(date +"%Y%m%d") pkgver=$(get_pkg_info pkgver) pkgrel=$(get_pkg_info pkgrel) bm_name=$(check_shared_pkg) -cgz_name="${bm_name}-${pkgver:-0}-${pkgrel:-0}_${date}.cgz" +cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy @@ -98,8 +98,8 @@ update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return - ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/${bm_name}.cgz" || return - echo "create package: $sync_dest/${bm_name}.cgz -> $(realpath $sync_dest/$cgz_name)" + ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/latest.cgz" || return + echo "create package: $sync_dest/latest.cgz -> $(realpath $sync_dest/$cgz_name)" update_shared_pkg "${bm_name}.cgz" -- 2.23.0

1 0

[PATCH compass-ci] container: fix failed to build archlinux images
by Liu Yinsi 02 Nov '20

02 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 becaue archlinux mirror not support x86_64 machine. [how] when deploy in x86_64 machine, change mirror http://mirrors.tuna.tsinghua.edu.cn/archlinuxarm/$arch/$repo => http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/build | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..9194f4e 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[ "$(arch)" == "x86_64" ] && + sed -i 's/archlinuxarm\/$arch\/$repo/archlinux\/$repo\/os\/$arch/g' root/etc/pacman.d/mirrorlist + docker build -t archlinux:testbed . -- 2.23.0

2 2

[PATCH compass-ci 2/2] container/assign-account: answerback-email.rb
by Luan Shengde 02 Nov '20

02 Nov '20

1. disable use golbal variable apply_info [why] Style/GlobalVars: Do not introduce global variables. [how] use local variables instead of global variables 2. enable transfer user info when execute apply account command [why] when applying account, the assign-account service will write the user info to user's default config file ~/.config/compass-ci/default/${USER}.yaml user info: - my_email - my_name - my_uuid - lab [how] transfer the user info along with the pub_key to assign-account service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 60 ++++++++++++-------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..4227081 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -18,29 +18,31 @@ names = Set.new %w[ JUMPER_PORT SEND_MAIL_HOST_INTERNET SEND_MAIL_PORT_INTERNET + LAB ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = defaults['LAB'] || z9 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, 'my_ssh_pubkey' => nil } -def init_info(email_file) - mail_content = Mail.read(email_file) +def init_info(mail_content, apply_info) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded.gsub(/\r|\n/, '') + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| @@ -52,15 +54,18 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + apply_info['my_email'] = email_address + # when apply account with email address, will not get the user name + apply_info['my_name'] = '' end opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + mail_content = Mail.read(email_file) + init_info(mail_content, apply_info) end opts.on_tail('-h|--help', 'show this message') do @@ -94,26 +99,31 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def account_info(user_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{user_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if apply_info['my_email'].nil? + + user_info = { + 'my_email': apply_info['my_email'], + 'my_name': apply_info['my_name'], + # for manually assign account, there will be no my_uuid specified + 'my_uuid': '', + 'lab': LAB, + 'pub_key': apply_info['my_ssh_pubkey'] + } - acct_info = account_info($apply_info['my_ssh_pubkey']) + acct_info = account_info(user_info) - message = build_message($apply_info['my_email'], acct_info) + message = build_message(apply_info['my_email'], acct_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +send_account(apply_info) -- 2.23.0

4 6

[PATCH v6 compass-ci 3/3] kernel_version.md: explain key "kernel_version"
by Xu Xijian 02 Nov '20

02 Nov '20

[why] Explain the meaning of new key "kernel_version" for scheduler, including what's the mapping between its typical values and actual files in the disk and how to set it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 doc/job/kernel_version.md diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md new file mode 100644 index 0000000..56a6b1a --- /dev/null +++ b/doc/job/kernel_version.md @@ -0,0 +1,32 @@ +# kernel_version + +Meaning: +- Every os has its kernel, however an os can start with different kernels according to different need. +- kernel_version is a key for users to specify a kernel version. +- If kernel_version is not given by users, it will use the default one. + +Related files: +- In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. +- File paths like below, An example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot" + +- $boot_dir/vmlinuz => $boot_dir/vmlinuz-$kernel_version + - default kernel file(vmlinuz) is a soft link + +- $boot_dir/modules => $boot_dir/modules-$kernel_version.cgz + - default modules is a soft link + +- $boot_dir/headers => $boot_dir/headers-$kernel_version.cgz + - default headers is a soft link + +- $boot_dir/vmlinuz-$kernel_version + - real kernel path + +- $boot_dir/modules-$kernel_version.cgz + - real modules path + +- $boot_dir/headers-$kernel_version.cgz + - real headers path + + +Usage example: +- submit iperf.yaml testbox=vm-hi1620-2p8g--xxj os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

2 2

[PATCH compass-ci 1/2] container/assign-account: get_account_info add new function
by Luan Shengde 02 Nov '20

02 Nov '20

[why]: esaier for user to config the default yaml file [how]: when get request data, parse the data and write the user infos to default yaml file: ~/.config/compass-ci/default/${USER}.yaml include: - my_email - my_name - my_uuid - lab Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/get_account_info.rb | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..4e61bdc 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -111,9 +113,23 @@ class AccountStorage } setup_authorized_key(account, pub_key) + setup_default_yaml(account) if @data.key? 'my_email' + return jumper_account_info end + def setup_default_yaml(account) + default_yaml_dir = File.join('/home', account, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + File.open("#{default_yaml_dir}/#{account}.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" if @data.key? 'my_name' + file.puts "my_uuid: #{@data['my_uuid']}" if @data.key? 'my_uuid' + file.puts "lab: #{@data['lab']}" if @data.key? 'lab' + file.close + end + end + def setup_authorized_key(account, pub_key) ssh_dir = File.join('/home/', account, '.ssh') Dir.mkdir ssh_dir, 0o700 -- 2.23.0

6 10

[PATCH lkp-tests] tests/*: fix cci job dir layout
by Wang Yong 02 Nov '20

02 Nov '20

[Why] for multi version support, cgz file for cci-makepkg need change dir layout, and change cci-depends dir layout together to easily update scheduler code path: /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/ => /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/${pkgname}/ filename: cci-makepkg ${pkgver}-${pkgrel}.cgz latest.cgz -> ${pkgver}-${pkgrel}.cgz filename: cci-depends ${benchmark}.cgz latest.cgz -> ${benchmark}.cgz E.g. output: cci-depends $ tree iperf ├── iperf.cgz └── latest.cgz -> iperf.cgz output: cci-makepkg $ tree sar ├── git-1.cgz └── latest.cgz -> git-1.cgz Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 7 +++---- distro/clear | 7 +++---- distro/debian | 8 +++----- tests/cci-depends | 4 ++-- tests/cci-makepkg | 13 ++++++------- 5 files changed, 17 insertions(+), 22 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 79656f0e4..a6ca49968 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -80,7 +80,6 @@ yum_download() yum_pack() { local target_dir=/opt/rpms - local date=$(date +"%Y%m%d") local downloaded_rpms=$(find /var/ -type f -name "*.rpm") @@ -90,10 +89,10 @@ yum_pack() mv $downloaded_rpms $target_dir - find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}.cgz - ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return - chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + ln -sf ${BM_NAME}.cgz $pack_to/latest.cgz || return + chown .lkp $pack_to/${BM_NAME}.cgz $pack_to/latest.cgz || return echo package installed to $pack_to/${BM_NAME}.cgz ls $target_dir/*.rpm > $pack_to/.${BM_NAME}.packages diff --git a/distro/clear b/distro/clear index f21de89c3..3b562aac6 100755 --- a/distro/clear +++ b/distro/clear @@ -121,7 +121,6 @@ list_bundles_files() pack_bundles_files() { - local date=$(date +"%Y%m%d") # allow empty ls $tmpdir | grep -q "\.list$" >/dev/null || { echo "empty PACKAGE_LIST for $BM_NAME" @@ -130,10 +129,10 @@ pack_bundles_files() cat $tmpdir/*.list | sort | uniq | cpio --quiet -o -H newc --owner=root.root | - gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz || return + gzip -n -9 >$pack_to/${BM_NAME}.cgz || return - ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return - chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + ln -sf ${BM_NAME}.cgz $pack_to/latest.cgz || return + chown .lkp $pack_to/${BM_NAME}.cgz $pack_to/latest.cgz || return echo package installed to $pack_to/${BM_NAME}.cgz } diff --git a/distro/debian b/distro/debian index 13c4e983a..270f77e01 100755 --- a/distro/debian +++ b/distro/debian @@ -99,8 +99,6 @@ fixup_perf() pack() { - local date=$(date +"%Y%m%d") - [[ $(type -t fixup_$BM_NAME) = 'function' ]] && fixup_$BM_NAME { @@ -112,12 +110,12 @@ pack() cat $LKP_SRC/distro/$DISTRO/$BM_NAME-allowlist } | cpio --quiet -o -H newc --owner=root.root | - gzip -n -9 > $pack_to/${BM_NAME}_$date.cgz || return + gzip -n -9 > $pack_to/${BM_NAME}.cgz || return - ln -sf ${BM_NAME}_$date.cgz \ + ln -sf ${BM_NAME}.cgz \ $pack_to/${BM_NAME}.cgz || return - chown .lkp $pack_to/${BM_NAME}_$date.cgz \ + chown .lkp $pack_to/${BM_NAME}.cgz \ $pack_to/${BM_NAME}.cgz || return echo package installed to $pack_to/${BM_NAME}.cgz diff --git a/tests/cci-depends b/tests/cci-depends index 36923ea4d..1c840a4ee 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -18,7 +18,7 @@ DEPS_MNT=/initrd/deps && mkdir -p "$DEPS_MNT" DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} pack_arch=$os_arch [[ "$benchmark" = "${benchmark##*.}" ]] || @@ -38,7 +38,7 @@ umask 002 } check_shared_package "$benchmark" && { - ln -sf $BM_NAME.cgz $pack_to/$benchmark.cgz + ln -sf latest.cgz $pack_to/$benchmark.cgz echo "create symlink for shared package: $benchmark -> $BM_NAME" exit 0 diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9d5082629..115136f61 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -26,7 +26,7 @@ mkdir -p "$PKG_MNT" export DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} . $LKP_SRC/distro/${DISTRO} . $LKP_SRC/lib/install.sh @@ -65,8 +65,8 @@ update_shared_pkg() [ "$bm_name" = "$benchmark" ] && return # benchmark is a symlink - ln -sf "$bm_link" "$sync_dest/${benchmark}.cgz" || return - echo "update shared pkg link ${benchmark}.cgz -> $bm_link" + ln -sf latest.cgz "$sync_dest/${benchmark}.cgz" || return + echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } distro_install_depends lkp-dev @@ -82,11 +82,10 @@ distro_install_depends ${benchmark}-dev build_depends_pkg -i $benchmark $TMP sync_dest="$PKG_MNT/$pack_to" -date=$(date +"%Y%m%d") pkgver=$(get_pkg_info pkgver) pkgrel=$(get_pkg_info pkgrel) bm_name=$(check_shared_pkg) -cgz_name="${bm_name}-${pkgver:-0}-${pkgrel:-0}_${date}.cgz" +cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy @@ -98,8 +97,8 @@ update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return - ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/${bm_name}.cgz" || return - echo "create package: $sync_dest/${bm_name}.cgz -> $(realpath $sync_dest/$cgz_name)" + ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/latest.cgz" || return + echo "create package: $sync_dest/latest.cgz -> $(realpath $sync_dest/$cgz_name)" update_shared_pkg "${bm_name}.cgz" -- 2.23.0

4 5

[PATCH compass-ci] area: fix src/lib/web_backend.rb "/compare_candidates" tbox_group regex error
by Zhang Yuhang 02 Nov '20

02 Nov '20

[error info] 1. /\d+$/ will be matched, such as "xxx123". 2. `index = "xxx123".index('--') || "xxx123".rindex('-')` - index equal nil. 3. `r = r[0, nil]` - raise a TypeError. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c3f6bef..55c5e4a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

2 2