- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] lib/job.rb: support more testbox formats
by Li Ping 16 Nov '20

16 Nov '20

If user wants to reproduce the result directly in the result directory. [problem] /srv/result/iperf/vm-2p8g/2020-11-09/z9.147520% submit job.yaml output: /home/liping/lkp-tests/lib/job.rb:344:in `get_hosts_file': hosts file not exist: vm-2p8g.taishan200-2280-2s64p-256g, maybe need check testbox field (ArgumentError) --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..6dd93f01 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -337,7 +337,7 @@ class Job lab_hosts_file = get_lab_hosts_file return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split('--')[0] + hosts_file_name = @job['tbox_group'].split(/(\.|--)/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" return hosts_file if File.file?(hosts_file) -- 2.23.0

2 2

[PATCH v4 compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 16 Nov '20

16 Nov '20

when execute 3-code/dev-env, error message as follows: ./dev-env: line 45: ./os/openEuler: No such file or directory no file ./os/openEuler is normal, will not affect deploy CCI, so fix it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 77027d9..9fcc1b4 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -42,4 +42,4 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +. $(dirname ${BASH_SOURCE[0]})/os/${ID} 2> /dev/null -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 15 Nov '20

15 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..1faa1e9f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,6 +37,23 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +check_pkgfile() +{ + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + + get_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 @@ -69,6 +86,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && check_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 1

[PATCH compass-ci] lib/es-query: fix NoMethodError
by Lu Weitao 14 Nov '20

14 Nov '20

[Why] Traceback (most recent call last): 4: from /c/compass-ci/sbin/compare:70:in `<main>' 3: from /c/compass-ci/lib/compare.rb:97:in `compare_by_template' 2: from /c/compass-ci/lib/compare.rb:128:in `create_groups_matrices' 1: from /c/compass-ci/lib/es_query.rb:21:in `multi_field_query' /c/compass-ci/lib/es_query.rb:54:in `build_mutli_field_subquery_body': undefined method `each' for nil:NilClass (NoMethodError) [Root cause] when (sbin/compare) call es_query, may pass empty filter as input parameter [How] exit when the filter is empty Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 037213d..c027aa9 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,6 +18,10 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) def multi_field_query(items, size: 10_000) + unless items + warn 'empty filter!' + exit + end query_fields = build_mutli_field_subquery_body items query = { query: { -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/makepkg: download submodules code from LKP server
by Liu Shaofei 14 Nov '20

14 Nov '20

Some repositories have submodules. So when build packages, we need download submodules from LKP server, avoid the waste of time in downloading files from the remote end. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index fde82212..c8b2ed00 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -623,6 +623,11 @@ extract_git() { cd_safe "${dir##*/}" + # modify submodules address to local server address. + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> ~/.gitconfig + fi + local ref=origin/HEAD if [[ -n $fragment ]]; then case ${fragment%%=*} in -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: change the result root dir
by Cao Xueliang 14 Nov '20

14 Nov '20

The new result root dir: /srv/result/$suite/$submit_date/$tbox_group/$rootfs/$pp_params/$job_id Example: rootfs: openeuler-20.09-aarch64 ($os-$os_version-$os_aarch) pp_params: socket-process-50 (use "-" connect pp's value up to 40 size) Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 18 +++++++++-- src/scheduler/pp_params.cr | 61 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 src/scheduler/pp_params.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index be0fdbb..4bef329 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -8,6 +8,7 @@ require "any_merge" require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" +require "scheduler/pp_params.cr" struct JSON::Any def []=(key : String, value : String) @@ -78,6 +79,9 @@ class Job initrd_pkg initrd_deps initrds_uri + rootfs + pp_params + submit_date result_root access_key access_key_file @@ -135,6 +139,9 @@ class Job append_init_field() set_user_lkp_src() set_os_dir() + set_submit_date() + set_pp_params() + set_rootfs() set_result_root() set_result_service() set_os_mount() @@ -188,10 +195,17 @@ class Job self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end + private def set_submit_date + self["submit_date"] = Time.local.to_s("%F") + end + + private def set_rootfs + self["rootfs"] = "#{os}-#{os_version}-#{os_arch}" + end + def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group - date = Time.local.to_s("%F") - self["result_root"] = "/result/#{suite}/#{tbox_group}/#{date}/#{id}" + self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") # access_key has information based on "result_root" # so when set result_root, we need redo set_ to update it diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr new file mode 100644 index 0000000..28ec6ef --- /dev/null +++ b/src/scheduler/pp_params.cr @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + private def set_pp_params + temp_hash = Hash(String, JSON::Any).new + + if @hash["pp"] + flat_hash(@hash["pp"].as_h, temp_hash) + end + + self["pp_params"] = sort_keys_return_values(temp_hash) + end + + private def flat_hash(old_hash, new_hash) + old_hash.each do |key1, value1| + if value1.as_h? + next if value1.as_h.empty? + + temp_hash = Hash(String, JSON::Any).new + value1.as_h.each do |key2, value2| + temp_hash.merge!({"#{key1}-#{key2}" => value2}) + end + + flat_hash(temp_hash, new_hash) + else + new_hash.merge!({key1 => value1}) + end + end + end + + private def sort_keys_return_values(hash) + values = [] of String + items_size = 0 + + hash.keys.sort.each do |key| + value = format_string(hash[key].to_s) + next if 0 == value.size + + if 40 > value.size + items_size + values.size + values << value + items_size += value.size + else + break + end + end + + return values.join("-") + end + + private def format_string(original_str) + temp = [] of String + original_str.each_char do |char| + if "#{char}" =~ /-|\w/ + temp << "#{char}" + end + end + + return temp.join() + end +end -- 2.23.0

1 0

[PATCH compass-ci] sched: change the result root dir
by Cao Xueliang 14 Nov '20

14 Nov '20

The new result root dir: /srv/result/$suite/$submit_date/$tbox_group/$rootfs/$pp_params/$job_id Example: rootfs: openeuler-20.09-aarch64 ($os-$os_version-$os_aarch) pp_params: socket-process-50 (use "-" connect pp's value up to 40 size) Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 22 ++++++++++++-- src/scheduler/pp_params.cr | 61 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+), 2 deletions(-) create mode 100644 src/scheduler/pp_params.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index be0fdbb..f723100 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -8,6 +8,7 @@ require "any_merge" require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" +require "scheduler/pp_params.cr" struct JSON::Any def []=(key : String, value : String) @@ -78,6 +79,9 @@ class Job initrd_pkg initrd_deps initrds_uri + rootfs + pp_params + submit_date result_root access_key access_key_file @@ -135,6 +139,9 @@ class Job append_init_field() set_user_lkp_src() set_os_dir() + set_submit_date() + set_pp_params() + set_rootfs() set_result_root() set_result_service() set_os_mount() @@ -188,10 +195,21 @@ class Job self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end + private def set_submit_date + self["submit_date"] = Time.local.to_s("%F") + end + + private def set_rootfs + self["rootfs"] = "#{os}-#{os_version}-#{os_arch}" + end + def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group - date = Time.local.to_s("%F") - self["result_root"] = "/result/#{suite}/#{tbox_group}/#{date}/#{id}" + if pp_params != "" + self["result_root"] = "/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}/#{pp_params}/#{id}" + else + self["result_root"] = "/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}/#{id}" + end # access_key has information based on "result_root" # so when set result_root, we need redo set_ to update it diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr new file mode 100644 index 0000000..28ec6ef --- /dev/null +++ b/src/scheduler/pp_params.cr @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + private def set_pp_params + temp_hash = Hash(String, JSON::Any).new + + if @hash["pp"] + flat_hash(@hash["pp"].as_h, temp_hash) + end + + self["pp_params"] = sort_keys_return_values(temp_hash) + end + + private def flat_hash(old_hash, new_hash) + old_hash.each do |key1, value1| + if value1.as_h? + next if value1.as_h.empty? + + temp_hash = Hash(String, JSON::Any).new + value1.as_h.each do |key2, value2| + temp_hash.merge!({"#{key1}-#{key2}" => value2}) + end + + flat_hash(temp_hash, new_hash) + else + new_hash.merge!({key1 => value1}) + end + end + end + + private def sort_keys_return_values(hash) + values = [] of String + items_size = 0 + + hash.keys.sort.each do |key| + value = format_string(hash[key].to_s) + next if 0 == value.size + + if 40 > value.size + items_size + values.size + values << value + items_size += value.size + else + break + end + end + + return values.join("-") + end + + private def format_string(original_str) + temp = [] of String + original_str.each_char do |char| + if "#{char}" =~ /-|\w/ + temp << "#{char}" + end + end + + return temp.join() + end +end -- 2.23.0

2 2

[PATCH compass-ci] container/kibana: fix build error
by Wu Zhende 14 Nov '20

14 Nov '20

Sending build context to Docker daemon 5.12kB Step 1/4 : ARG BASE_IMAGE Step 2/4 : FROM BASE_IMAGE invalid reference format: repository name must be lowercase Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 6e0dba0..970eb5a 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -3,7 +3,7 @@ ARG BASE_IMAGE -FROM BASE_IMAGE +FROM ${BASE_IMAGE} # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/3] iso2rootfs: add some configurations after generated rootfs
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] The initial rootfs have the follow two problems: - no dns resolver; - selinux is opened; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4f7843d7f348..05274164f8c1 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -338,6 +338,27 @@ run_qcow2rootfs() ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" } + +############ config rootfs ############ +config_dns_resolver() +{ + cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" + nameserver 114.114.114.114 + nameserver 8.8.8.8 + EOF +} + +disable_selinux() +{ + sed -i 's/^SELINUX=.*/SELINUX=disabled/g' "${ROOTFS_DES_DIR}/etc/selinux/config" +} + +config_rootfs() +{ + config_dns_resolver + disable_selinux +} + ############ test rootfs ############ get_qemu_efi_fd() { @@ -499,6 +520,8 @@ main() config_qcow2rootfs run_qcow2rootfs + config_rootfs + test_rootfs submit_test_job_yaml -- 2.23.0

2 2

[PATCH lkp-tests] tests/cci-makepkg: update gcc for centos7 when use initramfs os
by Lin Jiaxin 13 Nov '20

13 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 67d055a4..bdb2128d 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -88,13 +88,22 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +centos_update_gcc() +{ + if $(grep -sqF "CentOS Linux release 7" /etc/centos-release ); then + [[ "$os_mount" = "initramfs" ]] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable + fi +} + [ -n "$PKGBUILD_TAG" ] && get_pkgfile distro_install_depends lkp-dev -[ $os = "centos" ] && -grep -sqF "CentOS Linux release 7" /etc/centos-release && -source /opt/rh/devtoolset-8/enable +[ $os = "centos" ] && centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

1 0