January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/3] container/auto-submit: add build depends
by Liu Yinsi 12 Jan '21

12 Jan '21

store auto submit info to ES in build script, so add build depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends new file mode 100644 index 0000000..8357fca --- /dev/null +++ b/container/auto-submit/build-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH compass-ci 2/3] container/auto-submit: fix failed to submit job
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user install compass-ci at their local server, store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..8b8cc8c 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -1,6 +1,14 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t alpine:auto-submit . +require "#{ENV['CCI_SRC']}/lib/build_my_info_client" +config_yaml = "/etc/compass-ci/account/autosubmit.yaml" +option = YAML.load_file(config_yaml) + +build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) +build_my_info.store_account_info + +system 'docker build -t alpine:auto-submit .' -- 2.23.0

1 0

[PATCH compass-ci 1/3] container/auto-submit: bind mount volume
by Liu Yinsi 12 Jan '21

12 Jan '21

lkp-tests/lib/job.rb: load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" Dir.glob(['/etc/compass-ci/defaults/*.yaml', "#{self_config_path}/defaults/*.yaml"]).each do |file| [why] load_self_config support /etc/compass/defaults/ and $HOME/.config/compass-ci/defaults/, if mount /etc/compass-ci/account, it can't load config, so use /home/lkp/.config/compass-ci/defaults. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/auto_submit: only submit one branch job
by Liu Shaofei 12 Jan '21

12 Jan '21

[why] As some repositories have many branches, the git-mirror will submit jobs for each branch. So the large number of jobs are submitted for the same repository. To avoid this problem, just master branch is submitted. If repository does not have master branch, the last branch is selected to submit job. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..950d1f0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -28,16 +28,29 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) + # just submit master branch, if not master branch, submit last one branch. + def get_branch_argvs(newrefs_info) + repo = { 'branch' => '', 'commit' => '' } + newrefs_info['new_refs']['heads'].each do |branch, commit_id| - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + repo['branch'] = branch.delete_prefix('refs/heads/') + repo['commit'] = commit_id - system(real_argvs.join(' ')) + break if repo['branch'] == 'master' end + repo + end + + def submit(newrefs_info, submit_argv) + real_argvs = Array.new(submit_argv) + + repo = get_branch_argvs(newrefs_info) + + real_argvs.push("upstream_branch=#{repo['branch']}") + real_argvs.push("upstream_commit=#{repo['commit']}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{repo['commit']}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + system(real_argvs.join(' ')) end def get_argvs(newrefs_info) -- 2.23.0

1 1

[PATCH v2 compass-ci 1/2] bin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 12 Jan '21

12 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 1

[PATCH v2 lkp-tests] tests/*: reduce parameters
by Wang Yong 12 Jan '21

12 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 8 ++++---- tests/cci-depends | 11 +++++++---- tests/cci-makepkg | 11 +++++++---- 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 98341e7ac..48c738f52 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -9,10 +9,10 @@ # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -# - os -# - os_arch -# - os_version -# - os_mount +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..487121454 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..4f5f4c2f2 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci] fix: unusable package name
by Wang Chenglong 12 Jan '21

12 Jan '21

[Error] Packing package. Please wait. /usr/bin/setup-image: line 41: ./tmp/$IMAGE_PACK: ambiguous redirect before: IMAGE_PAVK=' openeuler-20.03.cgz' after: IMAGE_PAVK=openeuler-20.03.cgz Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index ce51ba3..8a7b315 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -12,7 +12,7 @@ check_passwd_file "$root_pwd_file" load_docker_img "$1" echo $docker_name_tag IMAGE_NAME=${docker_name_tag%%:*} -IMAGE_PACK=${IMAGE_NAME##*/}.cgz +IMAGE_PACK=$(echo ${IMAGE_NAME##*/}.cgz) start=( docker run -- 2.23.0

1 0

[PATCH v2 compass-ci] container/defconfig: optimize
by Luan Shengde 12 Jan '21

12 Jan '21

[why] when submitting jobs, the job will read data from /etc/compass-ci/defaults/*.yaml. the read data is open for user, but there are some sensitive data in the yaml files, like email address, email auth code, my_token .etc. [how] move the sensitive to yaml file in another directory: /etc/compass-ci/account/*.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index d1e2af0..fd13312 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end diff --git a/container/defconfig.sh b/container/defconfig.sh index 410ed75..97d924d 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -8,7 +8,12 @@ load_cci_defaults() { shopt -s nullglob - for i in /etc/compass-ci/defaults/*.yaml $HOME/.config/compass-ci/defaults/*.yaml + yaml_file=( + /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/account/*.yaml + "$HOME"/.config/compass-ci/defaults/*.yaml + ) + for i in "${yaml_file[@]}" do create_yaml_variables "$i" done -- 2.23.0

1 0

[PATCH v5 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= `cat /sys/class/net/eth0/operstate` is up, network is ok, but actually network is unreachable, because routing is not ready, it can't get PUB_NIC and PUB_IP by ip route command, so put export_ip_mac in network_ok to ensure network is ok. test: /srv/result/iperf/2021-01-12/vm-2p8g/debian-sid-riscv64/tcp-30/z9.325217 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

1 0

[PATCH v4 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 93.956875] + cat /sys/class/net/eth0/operstate [ 93.960236] ++ route -n [ 93.968810] + route='Kernel IP routing table [ 93.988935] Destination GatewayGenmaskFlags Metric Ref Use Iface' [ 94.011625] + echo Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= `cat /sys/class/net/eth0/operstate` is up, but routing is not ready, it can't get PUB_NIC and PUB_IP by ip route command, so check export_ip_mac in network_ok to make ensure network is ok. test: /srv/result/iperf/2021-01-12/vm-2p8g/debian-sid-riscv64/tcp-30/z9.325217 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

1 0