- Compass-ci - mailweb.openeuler.org

[PATCH v5 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

We will use 'rspec spec/submit_spec.rb' to merge job.yaml, it support merging from external file by "<<: job.yaml" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 +++ .../template/template-defaults.yaml | 19 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 19 +++++++++++++++++++ spec/submit_spec.rb | 4 ++++ 5 files changed, 46 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/template/template-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..331cb470 --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: atomic + +atomic: diff --git a/spec/submit/merge_yaml/template/template-defaults.yaml b/spec/submit/merge_yaml/template/template-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/template/template-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 169d7812..d71ed891 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -48,4 +48,8 @@ describe 'submit job spec' do it 'separate yaml spec' do submit_job('separate_yaml') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH compass-ci] job.cr: delete redundant "#{}" for string
by Xu Xijian 26 Nov '20

26 Nov '20

JobHelper.service_path() will return string value, so "#{}" is unnecessary for it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 378fd00..aa7ffc3 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -366,14 +366,14 @@ class Job private def set_kernel_uri self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{linux_vmlinuz_path}")}" + JobHelper.service_path("#{linux_vmlinuz_path}") end private def common_initrds temp_initrds = [] of String temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz")}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz") temp_initrds << "#{SCHED_HTTP_PREFIX}/job_initrd_tmpfs/#{id}/job.cgz" return temp_initrds @@ -384,13 +384,13 @@ class Job osimage_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{osimage_dir}/current")}" + JobHelper.service_path("#{osimage_dir}/current") temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" + JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz") temp_initrds << "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path(self["linux_modules_initrd"])}" + JobHelper.service_path(self["linux_modules_initrd"]) temp_initrds << "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path(self["linux_headers_initrd"])}" + JobHelper.service_path(self["linux_headers_initrd"]) temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? @@ -402,7 +402,7 @@ class Job temp_initrds = [] of String temp_initrds << "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp")}" + JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp") return temp_initrds end -- 2.23.0

2 1

[PATCH v2 compass-ci 5/5] openeuler: build openeuler initramfs image with one command
by Wang Chenglong 26 Nov '20

26 Nov '20

[Why] Use the tool to create a new openeuler-$version initramfs image with one command, and we also can customize the pre-installed software by configuring ./packages-to-install. [How] We create a link button for the user to use. $CCI_SRC/rootfs/initramfs/openeuler/aarch64/${os_version}/build --- container/osimage/openeuler/run | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 container/osimage/openeuler/run diff --git a/container/osimage/openeuler/run b/container/osimage/openeuler/run new file mode 100755 index 0000000..dde9985 --- /dev/null +++ b/container/osimage/openeuler/run @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/openeuler/lib +. ${CCI_SRC}/container/defconfig.sh + +export_root_passwd +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/files-to-exclude:/tmp/files-to-exclude + -v $DIR/${image}.repo:/etc/yum.repos.d/openEuler.repo + -e ROOT_PASSWD=$ROOT_PASSWD + -e IMAGE_NAME=$IMAGE_NAME + $image + /root/bin/create-image +) + +load_docker_image +"${cmd[@]}" +cp_image_to_host +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

1 0

[PATCH v2 compass-ci 4/5] openeuler: generate the cgz file in container
by Wang Chenglong 26 Nov '20

26 Nov '20

Install packages, set rootfs, pack images package and other actions in docker. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 37 ++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/osimage/openeuler/create-image diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image new file mode 100755 index 0000000..7402b2b --- /dev/null +++ b/container/osimage/openeuler/create-image @@ -0,0 +1,37 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +setup_rootfs() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + rm -f /etc/machine-id /var/lib/dbus/machine-id + systemd-machine-id-setup +} + +setup_login() { + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + chmod 644 /tmp/${IMAGE_NAME} +} + +install_packages +setup_rootfs +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH v2 compass-ci 3/5] openeuler: the function executed in host
by Wang Chenglong 26 Nov '20

26 Nov '20

Generate the root password, download and load the Docker image, and copy the image package to the host. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/lib | 45 +++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/osimage/openeuler/lib diff --git a/container/osimage/openeuler/lib b/container/osimage/openeuler/lib new file mode 100755 index 0000000..507b5b5 --- /dev/null +++ b/container/osimage/openeuler/lib @@ -0,0 +1,45 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR | awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR | awk -F "/" '{print $NF}')" +docker_url="https://repo.openeuler.org/openEuler-${os_version}/docker_img/aarch64/openE…" +docker_tar_file="openEuler-${os_version}-docker.aarch64.tar.xz" +image=$(echo "${os_name}-${os_version,,}") +IMAGE_NAME="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +export_root_passwd() +{ + local root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_PASSWD=$(cat "$root_pwd_file") +} + +load_docker_image() +{ + if [[ "$os_version" == '20.09' || "$os_version" == '20.03-LTS' ]]; then + [ -f "$HOME/$docker_tar_file" ] || wget -O "$HOME/$docker_tar_file" "$docker_url" + fi + + docker load -i "$HOME/$docker_tar_file" + + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[ERROR] Pull docker image failed." + exit 1 + fi +} + +cp_image_to_host() +{ + docker cp -a init_docker:/tmp/${IMAGE_NAME} $HOME/ + echo "result: $(ls $HOME/${IMAGE_NAME})" +} -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec.rb: submit debug_yaml/job_fail.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to debug job.yaml, and after the task is failed, it is be waiting state. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/debug_yaml/job_fail.yaml | 6 +++++ .../debug_yaml/job_fail/job_fail-3000.yaml | 22 +++++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 spec/submit/debug_yaml/job_fail.yaml create mode 100644 spec/submit/debug_yaml/job_fail/job_fail-3000.yaml diff --git a/spec/submit/debug_yaml/job_fail.yaml b/spec/submit/debug_yaml/job_fail.yaml new file mode 100644 index 00000000..86967162 --- /dev/null +++ b/spec/submit/debug_yaml/job_fail.yaml @@ -0,0 +1,6 @@ +suite: atomic + +on_fail: + sleep: 3000 + +atomic: diff --git a/spec/submit/debug_yaml/job_fail/job_fail-3000.yaml b/spec/submit/debug_yaml/job_fail/job_fail-3000.yaml new file mode 100644 index 00000000..3d7ad9b5 --- /dev/null +++ b/spec/submit/debug_yaml/job_fail/job_fail-3000.yaml @@ -0,0 +1,22 @@ +--- +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + sleep: 3000 + atomic: + cpufreq_governor: +monitors: -- 2.23.0

2 1

[PATCH compass-ci 2/5] openeuler: the function executed in host
by Wang Chenglong 26 Nov '20

26 Nov '20

Generate the root password, download and load the Docker image, and copy the image package to the host. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/lib | 45 +++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/osimage/openeuler/lib diff --git a/container/osimage/openeuler/lib b/container/osimage/openeuler/lib new file mode 100755 index 0000000..cf291ea --- /dev/null +++ b/container/osimage/openeuler/lib @@ -0,0 +1,45 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +docker_url="https://repo.openeuler.org/openEuler-${os_version}/docker_img/aarch64/openE…" +docker_tar="openEuler-${os_version}-docker.aarch64.tar.xz" +image=$(echo "${os_name}-${os_version}" | tr '[A-Z]' '[a-z]') +IMAGE_NAME="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +export_root_passwd() { + local root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_PASSWD=$(cat "$root_pwd_file") +} + +pull_docker_image() +{ + if [ "$os_version" == '20.09' ]; then + [ -f "$HOME/$docker_tar" ] || wget -O "$HOME/$docker_tar" "$docker_url" + elif [ "$os_version" == '20.03-LTS' ]; then + [ -f "$HOME/$docker_tar" ] || wget -O "$HOME/$docker_tar" "$docker_url" + fi + + docker load -i "$HOME/$docker_tar" + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[ERROR] Pull docker image failed." + exit 1 + fi +} + +docker_cp_image() +{ + docker cp -a init_docker:/tmp/${IMAGE_NAME} $HOME/ + echo "result: $(ls $HOME/${IMAGE_NAME})" +} -- 2.23.0

4 6

[PATCH compass-ci 2/3] web_backend.rb: fix : compare_groups_matrices needs arg suites_list
by Li Yuanchao 26 Nov '20

26 Nov '20

compare_groups_matrices needs three args, but now only get two, lost suites_list. divide method to pass rubocop limit Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..90c8a58 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -147,6 +147,20 @@ def get_es_must(params) must end +def _get_group_matrices(query, dimension) + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + while matrices.empty? + from += size + break if from > total + + query[:from] = from + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + end + [matrices, suites_list] +end + def get_groups_matrices(conditions, dimension, must, size, from) must += build_mutli_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } @@ -166,27 +180,17 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - while matrices.empty? - from += size - break if from > total - - query[:from] = from - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - end - matrices + matrices, suites_list = _get_groups_matrices(query, dimension) end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 1

[PATCH compass-ci 1/3] container/git-mirror: fix that can't get group id of committer in z9
by Li Yuanchao 26 Nov '20

26 Nov '20

In z9, 'committer' is only a group, not a user, so command 'id -g committer' can't get id of committer group Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/start | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index e1d7d8f..a618ec3 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -10,13 +10,13 @@ DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" lkp = %x(id -u lkp) -committer = %x(id -g committer) +group_id = %x(cat /etc/group | grep '^committer' | awk -F ':' '{print $3}') cmd = %W[ docker run --restart=always --name git-mirror - -u #{lkp.to_i}:#{committer.to_i} + -u #{lkp.to_i}:#{group_id.to_i} -d -e CCI_SRC=#{DOCKER_CCI} -e REPO_SRC=#{DOCKER_REPO} -- 2.23.0

1 1

[PATCH compass-ci 3/3] web-backend: fix : wrong method name
by Li Yuanchao 26 Nov '20

26 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 90c8a58..88ca02a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -147,7 +147,7 @@ def get_es_must(params) must end -def _get_group_matrices(query, dimension) +def _get_groups_matrices(query, dimension) result = es_query(query) matrices, suites_list = combine_group_query_data(result, dimension) while matrices.empty? -- 2.23.0

1 0