- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] tests/host-info: fix host-info script running error
by Zhang Yu 15 Dec '20

15 Dec '20

[why] The ipmi command just run in physical machine, the vm not support use ipmi command [how] Add a judgment only physical machine run ipmi operation Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index 84f796dcb..0ab8e6c97 100755 --- a/tests/host-info +++ b/tests/host-info @@ -1,4 +1,5 @@ #!/bin/bash +. $LKP_SRC/lib/env.sh get_memory() { @@ -101,8 +102,10 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX + is_virt && return + is_docker && return local ip=$(ipmitool lan print | awk 'NR==3{print $4}') - [ -n "$ip" ] && echo "ipmi_ip:" $ip + [ -n "$ip" ] && echo "ipmi_ip: $ip" } -- 2.23.0

1 0

[PATCH v2 compass-ci 4/6] fetch-mail: Dockerfile
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/fetch-mail/Dockerfile diff --git a/container/fetch-mail/Dockerfile b/container/fetch-mail/Dockerfile new file mode 100644 index 0000000..6e553b9 --- /dev/null +++ b/container/fetch-mail/Dockerfile @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM centos +MAINTAINER Luan Shengde <shdluan(a)163.com> +ENV DEBIAN_FRONTEND noninteractive + +RUN useradd -u 1000 mailer +RUN apt-get update && \ + apt-get install -y procmail fetchmail + +COPY fetchmailrc.template /home/mailer/.fetchmailrc +COPY procmailrc.template /home/mailer/.procmailrc +COPY run.sh /home/mailer/ + +RUN chmod 600 /home/mailer/.fetchmailrc +RUN chmod 600 /home/mailer/.procmailrc +RUN chown -R mailer:mailer /home/mailer -- 2.23.0

1 1

[PATCH v2 compass-ci 6/6] fetch-mail: start file
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/start | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/fetch-mail/start diff --git a/container/fetch-mail/start b/container/fetch-mail/start new file mode 100755 index 0000000..98cbd8f --- /dev/null +++ b/container/fetch-mail/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'fetch-mail' + +names = Set.new %w[ + FETCH_MAIL_ADDRESS + FETCH_MAIL_AUTH_CODE + FETCH_MAIL_DIR + FETCH_MAIL_BOX +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + -u mailer + --name=fetch-mail + -d +] + env + %W[ + -v #{defaults['FETCH_MAIL_DIR']}:#{defaults['FETCH_MAIL_DIR']}:rw + -w /home/mailer/ + fetch-mail +] + +cmd += ['sh', '-c', './run.sh'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v2 compass-ci 5/6] fetch-mail: build file
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/fetch-mail/build diff --git a/container/fetch-mail/build b/container/fetch-mail/build new file mode 100755 index 0000000..e72bed8 --- /dev/null +++ b/container/fetch-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t fetch-mail:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 3/6] fetch-mail: runner for fetch-mail service
by Luan Shengde 15 Dec '20

15 Dec '20

config fetchmail and procmail run fetchmail command to fetch emails Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/run.sh | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/fetch-mail/run.sh diff --git a/container/fetch-mail/run.sh b/container/fetch-mail/run.sh new file mode 100644 index 0000000..d7f9f38 --- /dev/null +++ b/container/fetch-mail/run.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +function mail_conf { + sed -i "s|FETCH_MAIL_ADDRESS|${FETCH_MAIL_ADDRESS}|g" '.fetchmailrc' + sed -i "s|FETCH_MAIL_AUTH_CODE|${FETCH_MAIL_AUTH_CODE}|g" '.fetchmailrc' + sed -i "s|FETCH_MAIL_DIR|${FETCH_MAIL_DIR}|g" '.procmailrc' + sed -i "s|FETCH_MAIL_BOX|${FETCH_MAIL_BOX}|g" '.procmailrc' +} + +mail_conf + +fetchmail -d 100 -- 2.23.0

1 0

[PATCH v2 compass-ci 2/6] fetch-mail: config file for procmail
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/procmailrc.template | 25 ++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 container/fetch-mail/procmailrc.template diff --git a/container/fetch-mail/procmailrc.template b/container/fetch-mail/procmailrc.template new file mode 100644 index 0000000..9cea4bc --- /dev/null +++ b/container/fetch-mail/procmailrc.template @@ -0,0 +1,25 @@ +DROPPRIVS=yes +PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin +MAILDIR=FETCH_MAIL_DIR +MAILLIST=${MAILDIR} +DEFAULT=${MAILDIR}/FETCH_MAIL_BOX/ +ORGMAIL=${DEFAULT}/FETCH_MAIL_BOX/ +LOGFILE=${MAILDIR}/log/procmail # recommended + +UMASK=002 + +DISPLAY=:0 +XAUTHORITY=$HOME/.Xauthority + +# One often receives duplicate messages for cross-postings to mailing +# lists, or explicit replies to a post from you in addition to the +# mailing list message. +# +# The following recipe eliminates all duplicate messages. It tells +# formail(1) to keep a cache file of the Message-IDs of the most recent +# messages you've received. Since Message-IDs are guaranteed unique for +# distinct messages, they are ideally suited to weed out duplicates. + +# LOCKFILE=$MAILDIR/.msgid.lock +:0 Wh: $MAILDIR/.msgid.lock +| formail -D 655360 $MAILDIR/log/msgid -- 2.23.0

1 0

[PATCH v2 compass-ci 1/6] fetch-mail: config file for fetchmail
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/fetchmailrc.template | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/fetch-mail/fetchmailrc.template diff --git a/container/fetch-mail/fetchmailrc.template b/container/fetch-mail/fetchmailrc.template new file mode 100644 index 0000000..8c7e82b --- /dev/null +++ b/container/fetch-mail/fetchmailrc.template @@ -0,0 +1,5 @@ +poll imap.qq.com with proto IMAP +username "FETCH_MAIL_ADDRESS" +password "FETCH_MAIL_AUTH_CODE" +mda "/usr/bin/procmail -f %F -d %T" +nokeep -- 2.23.0

1 0

[PATCH lkp-tests] tests/host-info: fix host-info script running error
by Zhang Yu 15 Dec '20

15 Dec '20

[why] The ipmi command just run in physical machine, the vm not support use ipmi command [how] Add a judgment only physical machine run ipmi operation Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/host-info b/tests/host-info index 84f796dcb..f222a7287 100755 --- a/tests/host-info +++ b/tests/host-info @@ -1,5 +1,7 @@ #!/bin/bash +. $LKP_SRC/lib/env.sh + get_memory() { memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') @@ -101,6 +103,7 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX + [ is_virt || is_docker ] && return local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip:" $ip -- 2.23.0

3 4

[PATCH lkp-tests] lib/env.sh: make the change of is_virt
by Zhang Yu 15 Dec '20

15 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/env.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/env.sh b/lib/env.sh index 78cfd6f57..fc34aaa8d 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -69,7 +69,8 @@ is_virt() # run as root [ -n "$(virt-what)" ] else - grep -q -w hypervisor /proc/cpuinfo + grep -q -w hypervisor /proc/cpuinfo && return 1 + return 0 fi } -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix format
by Lin Jiaxin 15 Dec '20

15 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index aafc120e..dbde3e54 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,8 +2,8 @@ # - pkgbuild_repo # - upstream_repo # - upstream_commit +# - upstream_url # - PKGBUILD_TAG -# upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -- 2.23.0

1 0

[PATCH lkp-tests 3/3] use ENV LKP_SRC
by Liu Yinsi 15 Dec '20

15 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- install.sh | 2 +- sbin/install-dependencies.sh | 4 ++-- sbin/set-env.sh | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/install.sh b/install.sh index 4bcc1afb4..cea1654f2 100644 --- a/install.sh +++ b/install.sh @@ -1,6 +1,6 @@ #!/bin/bash -. lib/env.sh +. $LKP_SRC/lib/env.sh make diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 3436ee44a..7605a29ff 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,7 +1,7 @@ #!/bin/bash -. lib/env.sh -. sbin/set-env.sh +. $LKP_SRC/lib/env.sh +. $LKP_SRC/sbin/set-env.sh # choose install function base on # DISTRIBUTION diff --git a/sbin/set-env.sh b/sbin/set-env.sh index 27471bb56..cc7f90506 100755 --- a/sbin/set-env.sh +++ b/sbin/set-env.sh @@ -1,6 +1,6 @@ #!/bin/sh -. lib/env.sh +. $LKP_SRC/lib/env.sh write_shell_profile() { -- 2.23.0

1 0

[PATCH lkp-tests 2/3] chmod +x
by Liu Yinsi 15 Dec '20

15 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 0 sbin/set-env.sh | 0 2 files changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 sbin/install-dependencies.sh mode change 100644 => 100755 sbin/set-env.sh diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh old mode 100644 new mode 100755 diff --git a/sbin/set-env.sh b/sbin/set-env.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH v3 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 15 Dec '20

15 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 87 ++++++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 41 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..3110685 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -152,29 +152,20 @@ job yaml ��[YAML](http://yaml.org/YAM 1. �� os �� rootfs�� openEuler �� - �� docker �� rootfs - + 1) �� openEuler �� docker �� - + ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` + ``` + 2) {compass-ci}/container/docker-rootfs �� '.tar.xz' �� docker �� rootfs + ```bash + ./{compass-ci}/container/docker-rootfs/run openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs-20.03 + ``` - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� + + 1) �� openEuler �� qcow2 �� ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… ``` @@ -183,45 +174,59 @@ job yaml ��[YAML](http://yaml.org/YAM cd {compass-ci}/container/qcow2rootfs ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) + + 2. ��rootfs + 1. �� chroot �� rootfs (�� root ��) ```bash chroot openEuler-rootfs ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 2. �� + a. �� /sbin/init + > ��systemd�� + ```bash + ls -l /sbin/init + /sbin/init -> ../lib/systemd/systemd + ``` + b. �� /etc/localtime + > �� + ```bash + ls -l /etc/localtime + /etc/localtime -> ../usr/share/zoneinfo/Asia/Shanghai + ``` + c. �� docker �� osimage �� + > docker �� /.dockerenv �� + + 3. ��rootfs + ```bash + cd $rootfs + find . | coip -o -Hnewc |gzip -9 > $os_name.cgz + ``` + + 4. ��/�� + �� {compass-ci} �� rootfs ��/�� + ```bash + find /lib/modules/* | cpio -o -Hnewc | gzip -9 > /modules-$os_name.cgz + find /usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > /headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

2 8

[PATCH v3 compass-ci] lib/job.cr: optimize check account error message
by Wu Zhende 15 Dec '20

15 Dec '20

Like: submit iperf.yaml failed, got job_id=0, error: Failed to verify the account. Please refer to https://gitee.com/wu_fengguang/compass-ci/doc/manual/apply-account.md Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0083a7d..8ce8649 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -326,7 +326,8 @@ class Job end private def check_account_info - error_msg = "Failed to verify the account. Please check your configuration" + error_msg = "Failed to verify the account.\n" + error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" account_info = @es.get_account(self["my_email"]) raise account_info unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH v2 lkp-tests] distro/depends: add some depends
by Liu Yinsi 15 Dec '20

15 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- distro/depends/cci-deploy | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/distro/depends/cci-deploy b/distro/depends/cci-deploy index c797c1eea..2b1d7fe52 100644 --- a/distro/depends/cci-deploy +++ b/distro/depends/cci-deploy @@ -8,3 +8,7 @@ bridge-utils qemu dmidecode lockfile-progs +iproute2 +vim +ruby-dev +busybox -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends: add some depends
by Liu Yinsi 15 Dec '20

15 Dec '20

command ip used in sparrow/2-network/br0. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- distro/depends/cci-deploy | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/distro/depends/cci-deploy b/distro/depends/cci-deploy index c797c1eea..2b1d7fe52 100644 --- a/distro/depends/cci-deploy +++ b/distro/depends/cci-deploy @@ -8,3 +8,7 @@ bridge-utils qemu dmidecode lockfile-progs +iproute2 +vim +ruby-dev +busybox -- 2.23.0

1 1

[PATCH v2 compass-ci] lib/job.cr: optimize check account error message
by Wu Zhende 15 Dec '20

15 Dec '20

Like: submit iperf.yaml failed, got job_id=0, error: Failed to verify the account. Please refer to https://gitee.com/wu_fengguang/compass-ci/doc/manual/apply-account.md Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0083a7d..697e6ae 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -326,7 +326,8 @@ class Job end private def check_account_info - error_msg = "Failed to verify the account. Please check your configuration" + error_msg = "Failed to verify the account.\n" + error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/doc/manual/apply-account.md" account_info = @es.get_account(self["my_email"]) raise account_info unless account_info.is_a?(JSON::Any) -- 2.23.0

2 1

[PATCH v3 lkp-tests 2/2] lib/job.rb: fix duplicate default field
by Wei Jihui 15 Dec '20

15 Dec '20

input: /etc/commpass-ci/default/crystal.yaml lab: lab1 ~/.config/.compass-ci/default/*.yaml lab: lab2 original output: #! /etc/commpass-ci/default/crystal.yaml lab: lab2 #! ~/.config/.compass-ci/default/*.yaml now output: #! /etc/commpass-ci/default/crystal.yaml #! ~/.config/.compass-ci/default/*.yaml lab: lab2 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 9eac4001..42b008c7 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -405,6 +405,7 @@ class Job return false end if defaults.is_a?(Hash) && !defaults.empty? + @defaults.delete_if { |key, _| defaults.has_key?(key) } @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) @defaults.merge!(@overrides) -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] lib/job.rb: load config from lab yaml
by Wei Jihui 15 Dec '20

15 Dec '20

[why] my_token is per-lab account, so it will load my_token from ~/.config/compass-ci/include/lab/$lab.yaml and $lab is from ~/.config/compass-ci/defaults/*.yaml or /etc/compass-ci/defaults/*.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index b41d55a7..9eac4001 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -414,6 +414,17 @@ class Job true end + def load_self_config + self_config_path = "#{ENV['HOME']}/.config/compass-ci" + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + "#{self_config_path}/defaults/*.yaml"]).each do |file| + load_one_defaults(file, @job) + end + + lab_yaml = File.join(self_config_path, 'include/lab', "#{@defaults['lab']}.yaml") + load_one_defaults(lab_yaml, @job) + end + def load_defaults(first_time = true) if @job.include? :no_defaults merge_defaults first_time @@ -426,11 +437,7 @@ class Job @file_loaded ||= {} end - Dir.glob(["/etc/compass-ci/defaults/*.yaml", - "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| - load_one_defaults(file, @job) - end - + load_self_config i = include_files job = deepcopy(@job) job['___'] = nil -- 2.23.0

1 0

[PATCH v4 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 15 Dec '20

15 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 87 ++++++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 41 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..3110685 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -152,29 +152,20 @@ job yaml ��[YAML](http://yaml.org/YAM 1. �� os �� rootfs�� openEuler �� - �� docker �� rootfs - + 1) �� openEuler �� docker �� - + ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` + ``` + 2) {compass-ci}/container/docker-rootfs �� '.tar.xz' �� docker �� rootfs + ```bash + ./{compass-ci}/container/docker-rootfs/run openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs-20.03 + ``` - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� + + 1) �� openEuler �� qcow2 �� ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… ``` @@ -183,45 +174,59 @@ job yaml ��[YAML](http://yaml.org/YAM cd {compass-ci}/container/qcow2rootfs ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) + + 2. ��rootfs + 1. �� chroot �� rootfs (�� root ��) ```bash chroot openEuler-rootfs ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 2. �� + a. �� /sbin/init + > ��systemd�� + ```bash + ls -l /sbin/init + /sbin/init -> ../lib/systemd/systemd + ``` + b. �� /etc/localtime + > �� + ```bash + ls -l /etc/localtime + /etc/localtime -> ../usr/share/zoneinfo/Asia/Shanghai + ``` + c. �� docker �� osimage �� + > docker �� /.dockerenv �� + + 3. ��rootfs + ```bash + cd $rootfs + find . | coip -o -Hnewc |gzip -9 > $os_name.cgz + ``` + + 4. ��/�� + �� {compass-ci} �� rootfs ��/�� + ```bash + find /lib/modules/* | cpio -o -Hnewc | gzip -9 > /modules-$os_name.cgz + find /usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > /headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow: fix that 0-package/os/openEuler can only be executed once
by Yu Chuan 15 Dec '20

15 Dec '20

[Error Msg] ++ rpm -i lib64lockfile1-1.16-1-omv4002.aarch64.rpm lockfile-progs-0.1.18-5-omv4002.aarch64.rpm warning: lib64lockfile1-1.16-1-omv4002.aarch64.rpm: Header V4 RSA/SHA256 Signature, key ID bf81de15: NOKEY package lib64lockfile1-1.16-1.aarch64 is already installed file /usr/lib64/liblockfile.so.1.0 from install of lib64lockfile1-1.16-1.aarch64 conflicts with file from package lib64lockfile1-1.16-1.aarch64 package lockfile-progs-0.1.18-5.aarch64 is already installed file /usr/bin/lockfile-check from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-create from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-remove from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-touch from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-lock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-touchlock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-unlock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/os/openEuler | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 43e96ab8ab2e..5b32c0a23784 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,17 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +install_rpm() +{ + local pkg=$1 + local pkg_url=$2 + + rpm -qa | grep -q "$pkg" && return 0 + + wget $pkg_url + rpm -i $(basename $pkg_url) && rm -rf $(basename $pkg_url) +} + pkgs=( vim git @@ -24,8 +35,10 @@ pkgs=( yum install -y --skip-broken "${pkgs[@]}" -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… +install_rpm \ + "lib64lockfile1-1.16-1.$(arch)"\ + "http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas…" -rpm -i lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm -rm -rf *.rpm +install_rpm \ + "lockfile-progs-0.1.18-5.$(arch)"\ + "http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas…" -- 2.23.0

2 3

[PATCH compass-ci 1/2] sparrow: refactor the code for docker build all image concurrently
by Yu Chuan 15 Dec '20

15 Dec '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/4-docker/buildall | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index eb245008ad45..43a3a06e4222 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -8,18 +8,13 @@ export action=$1 build_depends() { local container=$1 - local should_wait=$2 for dep in $(cat $container/*-depends 2> /dev/null) do - build_depends $CONTAINER_PATH/$dep block_wait + build_depends $CONTAINER_PATH/$dep done - if [ -n "$should_wait" ]; then - do_one $container - else - do_one $container & - fi + do_one $container } do_one() @@ -44,7 +39,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir + build_depends $dir & done wait -- 2.23.0

3 3

[PATCH v3 compass-ci 1/2] mail-robot: use key my_token instead of my_uuid
by Luan Shengde 15 Dec '20

15 Dec '20

[why] key my_uuid is renamed to my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 31 +++++++++++++---------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 0fff3b5..aea1aa5 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -30,14 +30,14 @@ SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 # my_info: # - my_email # - my_name -# - my_uuid +# - my_token # my_ssh_pubkey # store_account_info # call ESClient to store my_info # my_info: # - my_email # - my_name -# - my_uuid +# - my_token # - my_commit_url # - my_login_name # - my_ssh_pubkey @@ -87,21 +87,28 @@ class ApplyAccount accountx_es.query_by_id(@my_info['my_email']) end + def build_apply_info(apply_info, my_account_es) + my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? + apply_info.update my_account_es + apply_info.update @my_info + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + @my_info.update apply_info + apply_info['is_update_account'] = true + apply_info + end + def apply_my_account my_account_es = read_my_account_es apply_info = {} if my_account_es - my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') - apply_info.update my_account_es - apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq - @my_info.update apply_info - apply_info['is_update_account'] = true + build_apply_info(apply_info, my_account_es) else - my_uuid = %x(uuidgen).chomp - @my_info['my_uuid'] = my_uuid + my_token = %x(uuidgen).chomp + @my_info['my_token'] = my_token apply_info.update @my_info end + apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -112,10 +119,6 @@ class ApplyAccount @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es end - def check_account_es - ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) - end - def store_account_info es = ESClient.new(index: 'accounts') es.put_source_by_id(@my_info['my_email'], @my_info) -- 2.23.0

2 2

[PATCH compass-ci] lib/job.cr: optimize check account error message
by Wu Zhende 15 Dec '20

15 Dec '20

Like: submit iperf.yaml failed, got job_id=0, error: Failed to verify the account. Please refer to https://gitee.com/wu_fengguang/compass-ci/doc/manual/apply-account.md Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0083a7d..7eb8498 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -326,7 +326,8 @@ class Job end private def check_account_info - error_msg = "Failed to verify the account. Please check your configuration" + error_msg = "Failed to verify the account. " + error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/doc/manual/apply-account.md" account_info = @es.get_account(self["my_email"]) raise account_info unless account_info.is_a?(JSON::Any) -- 2.23.0

2 1

[PATCH lkp-tests] iso2rootfs: fix `has_cmd: command not found` error
by Yu Chuan 15 Dec '20

15 Dec '20

[Error Msg] ++ has_cmd yum /lkp/lkp/src/tests/iso2rootfs: line 213: has_cmd: command not found Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 8625b6484d89..9f2a77de17c6 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -6,6 +6,8 @@ set -e umask 0002 +. $LKP_SRC/lib/env.sh + # allow sut os array # - if job's os not in this array, job won't execute ALLOW_OS=( -- 2.23.0

3 2

[PATCH v2 lkp-tests 1/2] lib/job.rb: load config from lab yaml
by Wei Jihui 15 Dec '20

15 Dec '20

[why] my_uuid is per-lab account, so it will load my_uuid from ~/.config/compass-ci/include/lab/$lab.yaml and $lab is from ~/.config/compass-ci/defaults/*.yaml or /etc/compass-ci/defaults/*.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index b41d55a7..9eac4001 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -414,6 +414,17 @@ class Job true end + def load_self_config + self_config_path = "#{ENV['HOME']}/.config/compass-ci" + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + "#{self_config_path}/defaults/*.yaml"]).each do |file| + load_one_defaults(file, @job) + end + + lab_yaml = File.join(self_config_path, 'include/lab', "#{@defaults['lab']}.yaml") + load_one_defaults(lab_yaml, @job) + end + def load_defaults(first_time = true) if @job.include? :no_defaults merge_defaults first_time @@ -426,11 +437,7 @@ class Job @file_loaded ||= {} end - Dir.glob(["/etc/compass-ci/defaults/*.yaml", - "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| - load_one_defaults(file, @job) - end - + load_self_config i = include_files job = deepcopy(@job) job['___'] = nil -- 2.23.0

2 1

[PATCH compass-ci 6/6] fetch-mail: start file
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/start | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/fetch-mail/start diff --git a/container/fetch-mail/start b/container/fetch-mail/start new file mode 100755 index 0000000..372847b --- /dev/null +++ b/container/fetch-mail/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'fetch-mail' + +names = Set.new %w[ + FETCH_MAIL_ADDRESS + FETCH_MAIL_AUTH_CODE + FETCH_MAIL_DIR + FETCH_MAIL_BOX +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + -u mailer + --name=fetch-mail + -d +] + env + %W[ + -v #{defaults['FETCH_MAIL_DIR']}:#{defaults['FETCH_MAIL_DIR']}:rw + -w /home/mailer/ + fetch-mail +] + +cmd += ['sh', '-c', './run.sh'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci 5/6] fetch-mail: build file
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/fetch-mail/build diff --git a/container/fetch-mail/build b/container/fetch-mail/build new file mode 100755 index 0000000..e72bed8 --- /dev/null +++ b/container/fetch-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t fetch-mail:latest . -- 2.23.0

1 0

[PATCH compass-ci 4/6] fetch-mail: Dockerfile
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/fetch-mail/Dockerfile diff --git a/container/fetch-mail/Dockerfile b/container/fetch-mail/Dockerfile new file mode 100644 index 0000000..6e553b9 --- /dev/null +++ b/container/fetch-mail/Dockerfile @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM centos +MAINTAINER Luan Shengde <shdluan(a)163.com> +ENV DEBIAN_FRONTEND noninteractive + +RUN useradd -u 1000 mailer +RUN apt-get update && \ + apt-get install -y procmail fetchmail + +COPY fetchmailrc.template /home/mailer/.fetchmailrc +COPY procmailrc.template /home/mailer/.procmailrc +COPY run.sh /home/mailer/ + +RUN chmod 600 /home/mailer/.fetchmailrc +RUN chmod 600 /home/mailer/.procmailrc +RUN chown -R mailer:mailer /home/mailer -- 2.23.0

1 0

[PATCH compass-ci 3/6] fetch-mail: runner for fetch-mail service
by Luan Shengde 15 Dec '20

15 Dec '20

re-edit config file for fetchmail and procmail run fetchmail command to fetch emails Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/run.sh | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/fetch-mail/run.sh diff --git a/container/fetch-mail/run.sh b/container/fetch-mail/run.sh new file mode 100644 index 0000000..d7f9f38 --- /dev/null +++ b/container/fetch-mail/run.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +function mail_conf { + sed -i "s|FETCH_MAIL_ADDRESS|${FETCH_MAIL_ADDRESS}|g" '.fetchmailrc' + sed -i "s|FETCH_MAIL_AUTH_CODE|${FETCH_MAIL_AUTH_CODE}|g" '.fetchmailrc' + sed -i "s|FETCH_MAIL_DIR|${FETCH_MAIL_DIR}|g" '.procmailrc' + sed -i "s|FETCH_MAIL_BOX|${FETCH_MAIL_BOX}|g" '.procmailrc' +} + +mail_conf + +fetchmail -d 100 -- 2.23.0

1 0

[PATCH compass-ci 2/6] fetch-mail: config file for procmail
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/procmailrc.template | 25 ++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 container/fetch-mail/procmailrc.template diff --git a/container/fetch-mail/procmailrc.template b/container/fetch-mail/procmailrc.template new file mode 100644 index 0000000..9cea4bc --- /dev/null +++ b/container/fetch-mail/procmailrc.template @@ -0,0 +1,25 @@ +DROPPRIVS=yes +PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin +MAILDIR=FETCH_MAIL_DIR +MAILLIST=${MAILDIR} +DEFAULT=${MAILDIR}/FETCH_MAIL_BOX/ +ORGMAIL=${DEFAULT}/FETCH_MAIL_BOX/ +LOGFILE=${MAILDIR}/log/procmail # recommended + +UMASK=002 + +DISPLAY=:0 +XAUTHORITY=$HOME/.Xauthority + +# One often receives duplicate messages for cross-postings to mailing +# lists, or explicit replies to a post from you in addition to the +# mailing list message. +# +# The following recipe eliminates all duplicate messages. It tells +# formail(1) to keep a cache file of the Message-IDs of the most recent +# messages you've received. Since Message-IDs are guaranteed unique for +# distinct messages, they are ideally suited to weed out duplicates. + +# LOCKFILE=$MAILDIR/.msgid.lock +:0 Wh: $MAILDIR/.msgid.lock +| formail -D 655360 $MAILDIR/log/msgid -- 2.23.0

1 0

[PATCH compass-ci 1/6] fetch-mail: config file for fetchmail
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/fetchmailrc.template | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/fetch-mail/fetchmailrc.template diff --git a/container/fetch-mail/fetchmailrc.template b/container/fetch-mail/fetchmailrc.template new file mode 100644 index 0000000..8c7e82b --- /dev/null +++ b/container/fetch-mail/fetchmailrc.template @@ -0,0 +1,5 @@ +poll imap.qq.com with proto IMAP +username "FETCH_MAIL_ADDRESS" +password "FETCH_MAIL_AUTH_CODE" +mda "/usr/bin/procmail -f %F -d %T" +nokeep -- 2.23.0

1 0

[PATCH lkp-tests] lib/env.sh: modify the file that writes the variable
by Hu Xuejiao 15 Dec '20

15 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/env.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/env.sh b/lib/env.sh index 78cfd6f5..8ac42a83 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -122,6 +122,6 @@ is_system() shell_profile() { local profile="$HOME/.${SHELL##*/}rc" - [ -f "$profile" ] || profile="$HOME/.${SHELL##*/}_profile" + [ -f "$profile" ] || touch $profile echo $profile } -- 2.23.0

2 1

[PATCH lkp-tests] distro/depends: add initramfs-common
by Zhang Yu 15 Dec '20

15 Dec '20

When run job with initramfs, some dependency packages are reuired Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/initramfs-common | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/initramfs-common diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common new file mode 100644 index 000000000..15336b55a --- /dev/null +++ b/distro/depends/initramfs-common @@ -0,0 +1,2 @@ +time +perf -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/build-my-info.rb: building my_info from configuration files
by Liu Yinsi 15 Dec '20

15 Dec '20

before: users build info by input my_name, my_email, my_uuid, then write into ES and yaml file. after: no need users to input, get defaults administrator account info from configuration files /etc/compass-ci/account/*.yaml, then write into ES. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/build-my-info.rb | 54 +++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 73f7390..bb6e8d3 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -3,48 +3,46 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require 'json' require 'fileutils' require 'io/console' require_relative '../lib/es_client' - -print 'email: ' -my_email = $stdin.echo = gets.chomp -print 'name: ' -my_name = $stdin.echo = gets.chomp -my_uuid = %x(uuidgen).chomp - +require_relative '../container/defconfig' + +names = Set.new %w[ + my_email + my_name + my_uuid +] + +defaults = relevant_defaults(names) +my_email = defaults['my_email'] +my_name = defaults['my_name'] +my_uuid = defaults['my_uuid'] my_info = { 'my_email' => my_email, 'my_name' => my_name, - 'my_uuid' => my_uuid + 'my_uuid' => my_uuid, + 'my_login_name' => '', + 'my_commit_url' => '', + 'my_ssh_pubkey' => [] } -def store_account_info(my_info) - es = ESClient.new(index: 'accounts') - es.put_source_by_id(my_info['my_email'], my_info) +def es + ESClient.new(index: 'accounts') end -def config_default_yaml(my_info) - yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" - FileUtils.mkdir_p yaml_dir unless File.directory? yaml_dir - yaml_file = "#{yaml_dir}/account.yaml" - FileUtils.touch(yaml_file) unless File.exist? yaml_file - - yaml_info = YAML.load_file(yaml_file) || {} - yaml_info.update my_info +def store_account_info(my_info) + return if check_account_exist(my_info['my_email']) - File.open(yaml_file, 'w') do |f| - f.puts yaml_info.to_yaml - end + es.put_source_by_id(my_info['my_email'], my_info) end -def complete_my_info(my_info) - my_info['my_login_name'] = '' - my_info['my_commit_url'] = '' - my_info['my_ssh_pubkey'] = [] +def check_account_exist(email) + return true if es.query_by_id(email) + + return false end -config_default_yaml(my_info) -complete_my_info(my_info) store_account_info(my_info) -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/2] lib/job.rb: fix duplicate default field
by Wei Jihui 15 Dec '20

15 Dec '20

input: /etc/commpass-ci/default/crystal.yaml lab: lab1 ~/.config/.compass-ci/default/*.yaml lab: lab2 original output: #! /etc/commpass-ci/default/crystal.yaml lab: lab2 #! ~/.config/.compass-ci/default/*.yaml now output: #! /etc/commpass-ci/default/crystal.yaml #! ~/.config/.compass-ci/default/*.yaml lab: lab2 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 9eac4001..781a24a4 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -405,6 +405,7 @@ class Job return false end if defaults.is_a?(Hash) && !defaults.empty? + @defaults.delete_if { |key, _| defaults.keys.include?(key) } @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) @defaults.merge!(@overrides) -- 2.23.0

2 1

[PATCH compass-ci] container/defconfig.rb: fix rubocop warning
by Liu Yinsi 15 Dec '20

15 Dec '20

W: Lint/UselessAssignment: Useless assignment to variable - kb. Use _ or _kb as a variable name to indicate that it won't be used. File.readlines('/proc/meminfo').map { |s| k, v, kb = s.split; { k.chomp(':') => v } } ^^ C: Style/Semicolon: Do not use semicolons to terminate expressions. File.readlines('/proc/meminfo').map { |s| k, v, kb = s.split; { k.chomp(':') => v } } ^ Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 482dbc8..d1e2af0 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,11 +35,11 @@ def docker_rm(container) end def meminfo_hash - File.readlines('/proc/meminfo').map { |s| k, v, kb = s.split; { k.chomp(':') => v } } + YAML.load_file('/proc/meminfo') end def get_available_memory - memtotal = meminfo_hash[0]['MemTotal'].to_f / 1048576 + memtotal = meminfo_hash['MemTotal'].to_f / 1048576 # set container available memory size, minimum size is 1024m, maximum size is 30720m, # take the middle value according to the system memory size. -- 2.23.0

1 0

[PATCH lkp-tests] install missing dependent packages in x86_64
by Hu Xuejiao 15 Dec '20

15 Dec '20

[Error] /usr/lib/ld: cannot find -lc Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- install.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/install.sh b/install.sh index 4bcc1afb..c01d56f6 100644 --- a/install.sh +++ b/install.sh @@ -2,6 +2,8 @@ . lib/env.sh +has_cmd yum && yum install -y glibc-static + make source $(shell_profile) -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootfs/setup-image: set the system time zone
by Wang Chenglong 15 Dec '20

15 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 8e2e88a..787a42c 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -13,6 +13,11 @@ remove_file() rm -rf /.dockerenv } +set_localtime() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + setup_login() { [ -n "$ROOT_PASSWD" ] || return @@ -32,5 +37,6 @@ pack_cgz() install_packages remove_file +set_localtime setup_login pack_cgz -- 2.23.0

1 0

Re: [PATCH compass-ci] sched: replace access_key to upload_dirs
by Ren Wen 15 Dec '20

15 Dec '20

>- private def set_access_key >- self["access_key"] = "#{Random::Secure.hex(10)}-#{id}" >- self["access_key_file"] = File.join("/srv/", "#{result_root}", ".#{access_key}") >+ def set_upload_dirs >+ if @hash["cci-makepkg"]? >+ package_dir = ",/initrd/pkg" >+ elsif @hash["cci-depends"]? >+ package_dir = ",/initrd/deps" >+ elsif @hash["build-pkg"]? >+ package_dir = ",/initrd/build-pkg" >+ else >+ package_dir = "" >+ end The value of 'package_dir' doesn't need ',', Use: "#{result_root},#{package_dir}" instead. Thanks, RenWen >+ >+ self["upload_dirs"] = "#{result_root}#{package_dir}" > end > > private def set_result_service >diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr >index 579c2c0..d8f80d7 100644 >--- a/src/scheduler/close_job.cr >+++ b/src/scheduler/close_job.cr >@@ -8,8 +8,6 @@ class Sched > > job = @redis.get_job(job_id) > >- delete_access_key_file(job) if job >- > response = @es.set_job_content(job) > if response["_id"] == nil > # es update fail, raise exception >@@ -27,8 +25,4 @@ class Sched > > @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) > end >- >- def delete_access_key_file(job : Job) >- File.delete(job.access_key_file) if File.exists?(job.access_key_file) >- end > end >diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr >index 5a11523..6623a5e 100644 >--- a/src/scheduler/find_job_boot.cr >+++ b/src/scheduler/find_job_boot.cr >@@ -141,10 +141,6 @@ class Sched > return response > end > >- def touch_access_key_file(job : Job) >- FileUtils.touch(job.access_key_file) >- end >- > private def get_boot_ipxe(job : Job) > response = "#!ipxe\n\n" > response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" >@@ -155,8 +151,12 @@ class Sched > return response > end > >+ def set_id2upload_dirs(job) >+ @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) >+ end >+ > def boot_content(job : Job | Nil, boot_type : String) >- touch_access_key_file(job) if job >+ set_id2upload_dirs(job) if job > > case boot_type > when "ipxe" >-- >2.23.0 >

2 1

[PATCH v2 lkp-tests] sbin/submit: fix -m when submit job failed
by Wu Zhende 15 Dec '20

15 Dec '20

[Why] When use "submit -m", if submit job failed, the monitor is still run. [Error] submit iperf.yaml failed, got job_id=0, error: Failed to verify the account. Please check your configuration query=>{"job_id":[]} connect to ws://localhost:11310/filter Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/submit | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/submit b/sbin/submit index 6766e704a..05b3e86dd 100755 --- a/sbin/submit +++ b/sbin/submit @@ -162,6 +162,7 @@ ARGV.each do |jobfile| job_ids << msg['job_id'].to_s puts("submit #{jobfile}, got job_id=#{msg['job_id']}") else + opt_monitor = false puts("submit #{jobfile} failed, got job_id=#{msg['job_id']}, error: #{msg['message']}") end end -- 2.23.0

1 0

[PATCH compass-ci 10/40] doc: remove CRLF
by Wu Zhende 15 Dec '20

15 Dec '20

From: Wu Fengguang <wfg(a)mail.ustc.edu.cn> Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-for-a-test-machine.en.md | 490 ++++++------- doc/manual/compare-test-results.en.md | 96 +-- doc/manual/submit-command-description.md | 416 +++++------ ...latform-to-test-open-source-projects.en.md | 142 ++-- doc/manual/view-task-results.en.md | 46 +- ...00\346\272\220\351\241\271\347\233\256.md" | 136 ++-- doc/tutorial.md | 674 +++++++++--------- 7 files changed, 1000 insertions(+), 1000 deletions(-) diff --git a/doc/manual/apply-for-a-test-machine.en.md b/doc/manual/apply-for-a-test-machine.en.md index 33ef47f..14dd064 100644 --- a/doc/manual/apply-for-a-test-machine.en.md +++ b/doc/manual/apply-for-a-test-machine.en.md @@ -1,245 +1,245 @@ -# Prerequisites - -Ensure that you have performed the following operations according to the [apply-for-an-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/mas…: - -- Send an email to apply for an account. -- Receive an email from compass-ci(a)qq.com. -- Configure the local environment. - -# Applying for a Test Machine (VM) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (VM). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170593 - query=>{"job_id":["z9.170593"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} - {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} - {"job_id": "z9.170593", "job_state": "boot"} - {"job_id": "z9.170593", "job_state": "download"} - {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - {"job_state":"running","job_id":"z9.170593"} - {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 06:44:18 CST 2020 - - System load: 0.83 - Processes: 107 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.18.156.13 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (VM) after use. - - ```shell - root@vm-2p8g ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - -# Applying for a Test Machine (Physical Machine) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (physical machine). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170594 - query=>{"job_id":["z9.170594"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} - {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} - {"job_id": "z9.170594", "job_state": "boot"} - {"job_id": "z9.170594", "job_state": "download"} - {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} - {"job_state":"running","job_id":"z9.170594"} - {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 14:51:59 CST 2020 - - System load: 1.31 - Processes: 1020 - Memory used: 5.1% - Swap used: 0.0% - Usage On: 3% - IP address: 172.168.178.48 - Users online: 1 - - - - root@taishan200-2280-2s64p-256g--a5 ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (physical machine) after use. - - ```shell - root@taishan200-2280-2s64p-256g--a5 ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - -# FAQ - -* How Do I Change the Duration of Keeping the Test Machine when Applying for It? - - ```shell - hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml - suite: borrow - testcase: borrow - - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. - hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml - # After changing the value, submit the request again. - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - ``` - -* Guide to the **submit** Command - - Reference: [submit Command Description.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/m… - -* What Are the testbox Options? - - For details about the testbox options, visit https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts. - - > ![](./../public_sys-resources/icon-note.gif) **Note** - > - > VM testbox: vm-xxx - > - > PM testbox: taishan200-2280-xxx - - > ![](./../public_sys-resources/icon-notice.gif) **Notice** - > -> - If the testbox of a physical machine ends with `--axx`, a physical machine is specified. If a task is already in the task queue of the physical machine, the borrow task you submitted will not be processed until the previous task in the queue is completed. - > - If the testbox of a physical machine does not end with `-axx`, no physical machine is specified. In this case, the borrow task you submitted will be immediately allocated to idle physical machines in the cluster for execution. - -* How Do I Borrow the Specified Operating System? - - For details about the supported `os`, `os_arch`, and `os_version`, see [os-os\_verison-os\_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/…. \ No newline at end of file +# Prerequisites + +Ensure that you have performed the following operations according to the [apply-for-an-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/mas…: + +- Send an email to apply for an account. +- Receive an email from compass-ci(a)qq.com. +- Configure the local environment. + +# Applying for a Test Machine (VM) + +1. Generate a local RSA private-public key pair. + + ```shell + hi684@account-vm ~% ssh-keygen -t rsa + Generating public/private rsa key pair. + Enter file in which to save the key (/home/hi684/.ssh/id_rsa): + Created directory '/home/hi684/.ssh'. + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /home/hi684/.ssh/id_rsa. + Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. + The key fingerprint is: + SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm + The key's randomart image is: + +---[RSA 2048]----+ + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + +----[SHA256]-----+ + hi684@account-vm ~% ls -hla .ssh + total 16K + drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . + drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. + -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa + -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub + ``` + +2. Select the YAML file as required. + + A **lkp-tests** folder is stored in each user directory `/home/${USER}`. + + ```shell + hi684@account-vm ~% cd lkp-tests/jobs + hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* + -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml + -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml + -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml + ``` + +3. Submit the YAML file and connect to the test machine (VM). + + ```shell + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml + submit borrow-1h.yaml, got job_id=z9.170593 + query=>{"job_id":["z9.170593"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} + {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} + {"job_id": "z9.170593", "job_state": "boot"} + {"job_id": "z9.170593", "job_state": "download"} + {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} + {"job_state":"running","job_id":"z9.170593"} + {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} + Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Thu Nov 26 06:44:18 CST 2020 + + System load: 0.83 + Processes: 107 + Memory used: 6.1% + Swap used: 0.0% + Usage On: 89% + IP address: 172.18.156.13 + Users online: 1 + + + + root@vm-2p8g ~# + ``` + + For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. + +4. Return the test machine (VM) after use. + + ```shell + root@vm-2p8g ~# reboot + Connection to 172.168.131.2 closed by remote host. + Connection to 172.168.131.2 closed. + hi684@account-vm ~/lkp-tests/jobs% + ``` + +# Applying for a Test Machine (Physical Machine) + +1. Generate a local RSA private-public key pair. + + ```shell + hi684@account-vm ~% ssh-keygen -t rsa + Generating public/private rsa key pair. + Enter file in which to save the key (/home/hi684/.ssh/id_rsa): + Created directory '/home/hi684/.ssh'. + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /home/hi684/.ssh/id_rsa. + Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. + The key fingerprint is: + SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm + The key's randomart image is: + +---[RSA 2048]----+ + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + +----[SHA256]-----+ + hi684@account-vm ~% ls -hla .ssh + total 16K + drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . + drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. + -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa + -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub + ``` + +2. Select the YAML file as required. + + A **lkp-tests** folder is stored in each user directory `/home/${USER}`. + + ```shell + hi684@account-vm ~% cd lkp-tests/jobs + hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* + -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml + -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml + -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml + ``` + +3. Submit the YAML file and connect to the test machine (physical machine). + + ```shell + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml + submit borrow-1h.yaml, got job_id=z9.170594 + query=>{"job_id":["z9.170594"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} + {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} + {"job_id": "z9.170594", "job_state": "boot"} + {"job_id": "z9.170594", "job_state": "download"} + {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} + {"job_state":"running","job_id":"z9.170594"} + {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} + Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Thu Nov 26 14:51:59 CST 2020 + + System load: 1.31 + Processes: 1020 + Memory used: 5.1% + Swap used: 0.0% + Usage On: 3% + IP address: 172.168.178.48 + Users online: 1 + + + + root@taishan200-2280-2s64p-256g--a5 ~# + ``` + + For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. + +4. Return the test machine (physical machine) after use. + + ```shell + root@taishan200-2280-2s64p-256g--a5 ~# reboot + Connection to 172.168.131.2 closed by remote host. + Connection to 172.168.131.2 closed. + hi684@account-vm ~/lkp-tests/jobs% + ``` + +# FAQ + +* How Do I Change the Duration of Keeping the Test Machine when Applying for It? + + ```shell + hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml + suite: borrow + testcase: borrow + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + sshd: + # sleep at the bottom + sleep: 1h + hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml + sleep: 1h + # Use the VIM editor to change the value of the sleep field. + hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml + # After changing the value, submit the request again. + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml + ``` + +* Guide to the **submit** Command + + Reference: [submit Command Description.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/m… + +* What Are the testbox Options? + + For details about the testbox options, visit https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts. + + > ![](./../public_sys-resources/icon-note.gif) **Note** + > + > VM testbox: vm-xxx + > + > PM testbox: taishan200-2280-xxx + + > ![](./../public_sys-resources/icon-notice.gif) **Notice** + > +> - If the testbox of a physical machine ends with `--axx`, a physical machine is specified. If a task is already in the task queue of the physical machine, the borrow task you submitted will not be processed until the previous task in the queue is completed. + > - If the testbox of a physical machine does not end with `-axx`, no physical machine is specified. In this case, the borrow task you submitted will be immediately allocated to idle physical machines in the cluster for execution. + +* How Do I Borrow the Specified Operating System? + + For details about the supported `os`, `os_arch`, and `os_version`, see [os-os\_verison-os\_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/…. diff --git a/doc/manual/compare-test-results.en.md b/doc/manual/compare-test-results.en.md index 8a7e5f7..5a5dd22 100644 --- a/doc/manual/compare-test-results.en.md +++ b/doc/manual/compare-test-results.en.md @@ -1,48 +1,48 @@ -# How Do I Compare Test Results? -The Compare feature is used to analyze the results of different jobs, show the performance waves and changes under different influencing factors, for users to analyze performance influencing factor. - -## url: https://compass-ci.openeuler.org/compare - -## options directions -### filters: - - suite: iperf, netperf, mysql, ... - - OS: openeuler 20.03, centos 7.6, ... - - os_arch: aarch64, x86 - - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... - we can combine the above options arbitrarily to limit compare scope. - choose at least one option as filter. - -### dimension - Dimension can select: os, os_version, os_arch, suite, tbox_group. - Within filter, we will compare all different job result by dimension - and keep other test conditions are same. - -## example: - filter: suite = iperf - dimension: os_version - - result: - os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same - - - 20.09 20.03 metric - -------------------- ------------------------------ ------------------------------ - fails:runs change fails:runs - | | | - 3:3 -100.0% 0:3 last_state.exit_fail - 3:3 -100.0% 0:3 last_state.is_incomplete_run - 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 - 3:3 -100.0% 0:3 stderr.perf_command_failed - - - - 20.09 20.03 metric - -------------------- ------------------------------ ------------------------------ - %stddev change %stddev - \ | \ - 4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps - 4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps - 94.82 -44.0% 53.10 boot-time.boot - 123.11 -58.4% 51.19 boot-time.idle - 0.00 0 4.87 boot-time.kernel_boot - 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 \ No newline at end of file +# How Do I Compare Test Results? +The Compare feature is used to analyze the results of different jobs, show the performance waves and changes under different influencing factors, for users to analyze performance influencing factor. + +## url: https://compass-ci.openeuler.org/compare + +## options directions +### filters: + - suite: iperf, netperf, mysql, ... + - OS: openeuler 20.03, centos 7.6, ... + - os_arch: aarch64, x86 + - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... + we can combine the above options arbitrarily to limit compare scope. + choose at least one option as filter. + +### dimension + Dimension can select: os, os_version, os_arch, suite, tbox_group. + Within filter, we will compare all different job result by dimension + and keep other test conditions are same. + +## example: + filter: suite = iperf + dimension: os_version + + result: + os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric + -------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric + -------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ + 4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps + 4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 diff --git a/doc/manual/submit-command-description.md b/doc/manual/submit-command-description.md index b0faf99..4ea431c 100644 --- a/doc/manual/submit-command-description.md +++ b/doc/manual/submit-command-description.md @@ -1,208 +1,208 @@ -# submit Command Description - -### Prerequisites - -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/本地…. - -### Purpose - -You can run the **submit** command to submit a test task. This command provides multiple options to help you submit tasks more flexibly. You can enter the **submit** command in the command line to view the help information and use the command flexibly as required. - -### Basic Usage - -The test task is submitted in a YAML file. You need to prepare the YAML file of the test task. This document uses **iperf.yaml** as an example. Run the following command to submit a test task: - -``` -submit iperf.yaml -``` - -The following message is displayed: - -```shell -hi8109@account-vm ~% submit iperf.yaml -submit iperf.yaml, got job_id=z9.173924 -``` - -The **testbox** field has been added to the **iperf.yaml** file shown in this document. If the YAML file does not contain this field, an error is reported: - -```shell -hi8109@account-vm ~% submit iperf.yaml -submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbox' -``` - -You can add the **testbox** field to the YAML file or run the following command: - -``` -submit iperf.yaml testbox=vm-2p8g -``` - -The value of the **testbox** field specifies the required test machine. You can run the `ls` command to view the available test machines in the `lkp-tests/hosts` path, as shown in the following figure: - -```shell -hi8109@account-vm ~/lkp-tests/hosts% ll -total 120K --rw-r--r--. 1 root root 76 2020-11-02 14:54 vm-snb --rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p8g --rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p4g --rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p1g --rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-1p1g --rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p8g --rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p4g --rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p1g --rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-1p1g --rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p8g-pxe --rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p8g --rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p4g-pxe --rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p4g --rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p1g-pxe --rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p1g --rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-1p1g-pxe --rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-1p1g --rw-r--r--. 1 root root 14 2020-11-02 14:54 taishan200-2280-2s64p-256g --rw-r--r--. 1 root root 497 2020-11-02 14:54 lkp-skl-d01 --rw-r--r--. 1 root root 574 2020-11-02 14:54 lkp-ivb-d04 --rw-r--r--. 1 root root 606 2020-11-02 14:54 lkp-ivb-d02 --rw-r--r--. 1 root root 384 2020-11-02 14:54 lkp-ivb-d01 --rw-r--r--. 1 root root 388 2020-11-02 14:54 lkp-hsw-d01 --rw-r--r--. 1 root root 385 2020-11-02 14:54 lkp-bdw-de1 --rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-8g --rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-4g --rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-2g --rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-1g --rw-r--r--. 1 root root 13 2020-11-02 14:54 2288hv5-2s64p --rw-r--r--. 1 root root 74 2020-11-02 14:54 vm-snb-i386 -``` - -> ![](./../public_sys-resources/icon-note.gif) **Note** -> -> Use the equal sign (=) to update the fields in the YAML file. The priority of the ***=*** varies with its position in the command line. -> -> * In the **submit iperf.yaml testbox=vm-2p8g** command, the **=** is defined after the YAML file. As a result, the priority of the **=** is higher than that of the YAML file. **testbox=vm-2p8g** overrides the **testbox** field defined in the YAML file. -> * In the **submit testbox=vm-2p8g iperf.yaml** command, the **=** is defined before the YAML file. As a result, the priority of **=** is lower than that of the YAML file. **testbox=vm-2p8g** does not override the **testbox** field defined in the YAML file. A value is assigned only when the YAML file does not contain the **testbox** field. - -### Advanced Usage - -The following figure shows the options of the **submit** command: - -```shell -hi8109@account-vm ~% submit -Usage: submit [options] job1.yaml job2.yaml ... - submit test jobs to the scheduler - -options: - -s, --set 'KEY: VALUE' add YAML hash to job - -o, --output DIR save job yaml to DIR/ - -a, --auto-define-files auto add define_files - -c, --connect auto connect to the host - -m, --monitor monitor job status: use -m 'KEY: VALUE' to add rule - --my-queue add to my queue -``` - -* **Usage of -s** - - You can use the **-s'KEY:VALUE'** parameter to update the key-value pair to the submitted task. An example is shown in the following figure: - - ``` - submit -s 'testbox: vm-2p8g' iperf.yaml - ``` - - * If the **iperf.yaml** file does not contain **testbox: vm-2p8g**, the field will be added to the submitted task. - * If the **iperf.yaml** file contains the **testbox** field but the value is not **vm-2p8g**, the value of **testbox** in the submitted task will be updated as **vm-2p8g**. - -* **Usage of -o** - - You can run the **-o DIR** command to save the generated YAML file to the specified directory **DIR**. An example is shown in the following figure: - - ``` - submit -o ~/iperf.yaml - ``` - - After the command is executed, the YAML file that has been processed by the **submit** command is generated in the specified directory. - -* **Usage of -a** - - If the **lkp-tests** on the client is changed in the test case, you need to use the **-a** option for adaptation. Synchronize the modification made in the **lkp-tests** on the client to the server, and generate a customized test script on the test machine. An example is shown in the following figure: - - ``` - submit -a iperf.yaml - ``` - -* **Usage of -m** - - You can use the **-m** parameter to enable the task monitoring function and print the status information during the task execution on the console. In this way, you can monitor the execution process of the test task in real time. An example is shown in the following figure: - - ``` - submit -m iperf.yaml - ``` - - The following information is displayed on the console: - - ```shell - hi8109@account-vm ~% submit -m iperf.yaml - submit iperf.yaml, got job_id=z9.173923 - query=>{"job_id":["z9.173923"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.173923","message":"","job_state":"submit","result_root":"/srv/result/iperf/2020-11-30/vm-2p8g/openeuler-20.03-aarch6 - {"job_id": "z9.173923", "result_root": "/srv/result/iperf/2020-11-30/vm-2p8g/openeuler-20.03-aarch64/tcp-30/z9.173923", "job_state - {"job_id": "z9.173923", "job_state": "boot"} - {"job_id": "z9.173923", "job_state": "download"} - {"time":"2020-11-30 20:28:16","mac":"0a-f5-9f-83-62-ea","ip":"172.18.192.21","job_id":"z9.173923","state":"running","testbox":"vm- - {"job_state":"running","job_id":"z9.173923"} - {"job_state":"post_run","job_id":"z9.173923"} - {"start_time":"2020-11-30 12:25:15","end_time":"2020-11-30 12:25:45","loadavg":"1.12 0.38 0.14 1/105 1956","job_id":"z9.173923"} - {"job_state":"finished","job_id":"z9.173923"} - {"job_id": "z9.173923", "job_state": "complete"} - {"time":"2020-11-30 20:28:54","mac":"0a-f5-9f-83-62-ea","ip":"172.18.192.21","job_id":"z9.173923","state":"rebooting","testbox":"v - {"job_id": "z9.173923", "job_state": "extract_finished"} - connection closed: normal - ``` - -* **Usage of -c** - - The **-c** parameter must be used together with the **-m** parameter to implement the automatic login function in the task of applying for a device. - - An example is shown in the following figure: - - ``` - submit -m -c borrow-1h.yaml - ``` - - After submitting a task of applying for a device, you will receive the returned login information, such as `ssh ip -p port`. After adding the **-c** parameter, you can log in to the executor without manually entering the SSH login command. - - The following information is displayed on the console: - - ```shell - hi8109@account-vm ~% submit -m -c borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.173925 - query=>{"job_id":["z9.173925"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.173925","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-30/vm-2p8g/openeuler-20.03-aarch - {"job_id": "z9.173925", "result_root": "/srv/result/borrow/2020-11-30/vm-2p8g/openeuler-20.03-aarch64/3600/z9.173925", "job_state" - {"job_id": "z9.173925", "job_state": "boot"} - {"job_id": "z9.173925", "job_state": "download"} - {"time":"2020-11-30 20:35:04","mac":"0a-24-5d-c8-aa-d0","ip":"172.18.101.4","job_id":"z9.173925","state":"running","testbox":"vm-2 - {"job_state":"running","job_id":"z9.173925"} - {"job_id": "z9.173925", "state": "set ssh port", "ssh_port": "50200", "tbox_name": "vm-2p8g.taishan200-2280-2s48p-256g--a52-7"} - Host 172.168.131.2 not found in /home/hi8109/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50200' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Mon Nov 30 12:32:04 CST 2020 - - System load: 0.50 - Processes: 105 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.17.0.1 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - You log in to the executor successfully. \ No newline at end of file +# submit Command Description + +### Prerequisites + +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/本地…. + +### Purpose + +You can run the **submit** command to submit a test task. This command provides multiple options to help you submit tasks more flexibly. You can enter the **submit** command in the command line to view the help information and use the command flexibly as required. + +### Basic Usage + +The test task is submitted in a YAML file. You need to prepare the YAML file of the test task. This document uses **iperf.yaml** as an example. Run the following command to submit a test task: + +``` +submit iperf.yaml +``` + +The following message is displayed: + +```shell +hi8109@account-vm ~% submit iperf.yaml +submit iperf.yaml, got job_id=z9.173924 +``` + +The **testbox** field has been added to the **iperf.yaml** file shown in this document. If the YAML file does not contain this field, an error is reported: + +```shell +hi8109@account-vm ~% submit iperf.yaml +submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbox' +``` + +You can add the **testbox** field to the YAML file or run the following command: + +``` +submit iperf.yaml testbox=vm-2p8g +``` + +The value of the **testbox** field specifies the required test machine. You can run the `ls` command to view the available test machines in the `lkp-tests/hosts` path, as shown in the following figure: + +```shell +hi8109@account-vm ~/lkp-tests/hosts% ll +total 120K +-rw-r--r--. 1 root root 76 2020-11-02 14:54 vm-snb +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p8g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p4g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p1g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-1p1g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p8g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p4g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p1g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-1p1g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p8g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p8g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p4g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p4g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p1g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p1g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-1p1g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-1p1g +-rw-r--r--. 1 root root 14 2020-11-02 14:54 taishan200-2280-2s64p-256g +-rw-r--r--. 1 root root 497 2020-11-02 14:54 lkp-skl-d01 +-rw-r--r--. 1 root root 574 2020-11-02 14:54 lkp-ivb-d04 +-rw-r--r--. 1 root root 606 2020-11-02 14:54 lkp-ivb-d02 +-rw-r--r--. 1 root root 384 2020-11-02 14:54 lkp-ivb-d01 +-rw-r--r--. 1 root root 388 2020-11-02 14:54 lkp-hsw-d01 +-rw-r--r--. 1 root root 385 2020-11-02 14:54 lkp-bdw-de1 +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-8g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-4g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-2g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-1g +-rw-r--r--. 1 root root 13 2020-11-02 14:54 2288hv5-2s64p +-rw-r--r--. 1 root root 74 2020-11-02 14:54 vm-snb-i386 +``` + +> ![](./../public_sys-resources/icon-note.gif) **Note** +> +> Use the equal sign (=) to update the fields in the YAML file. The priority of the ***=*** varies with its position in the command line. +> +> * In the **submit iperf.yaml testbox=vm-2p8g** command, the **=** is defined after the YAML file. As a result, the priority of the **=** is higher than that of the YAML file. **testbox=vm-2p8g** overrides the **testbox** field defined in the YAML file. +> * In the **submit testbox=vm-2p8g iperf.yaml** command, the **=** is defined before the YAML file. As a result, the priority of **=** is lower than that of the YAML file. **testbox=vm-2p8g** does not override the **testbox** field defined in the YAML file. A value is assigned only when the YAML file does not contain the **testbox** field. + +### Advanced Usage + +The following figure shows the options of the **submit** command: + +```shell +hi8109@account-vm ~% submit +Usage: submit [options] job1.yaml job2.yaml ... + submit test jobs to the scheduler + +options: + -s, --set 'KEY: VALUE' add YAML hash to job + -o, --output DIR save job yaml to DIR/ + -a, --auto-define-files auto add define_files + -c, --connect auto connect to the host + -m, --monitor monitor job status: use -m 'KEY: VALUE' to add rule + --my-queue add to my queue +``` + +* **Usage of -s** + + You can use the **-s'KEY:VALUE'** parameter to update the key-value pair to the submitted task. An example is shown in the following figure: + + ``` + submit -s 'testbox: vm-2p8g' iperf.yaml + ``` + + * If the **iperf.yaml** file does not contain **testbox: vm-2p8g**, the field will be added to the submitted task. + * If the **iperf.yaml** file contains the **testbox** field but the value is not **vm-2p8g**, the value of **testbox** in the submitted task will be updated as **vm-2p8g**. + +* **Usage of -o** + + You can run the **-o DIR** command to save the generated YAML file to the specified directory **DIR**. An example is shown in the following figure: + + ``` + submit -o ~/iperf.yaml + ``` + + After the command is executed, the YAML file that has been processed by the **submit** command is generated in the specified directory. + +* **Usage of -a** + + If the **lkp-tests** on the client is changed in the test case, you need to use the **-a** option for adaptation. Synchronize the modification made in the **lkp-tests** on the client to the server, and generate a customized test script on the test machine. An example is shown in the following figure: + + ``` + submit -a iperf.yaml + ``` + +* **Usage of -m** + + You can use the **-m** parameter to enable the task monitoring function and print the status information during the task execution on the console. In this way, you can monitor the execution process of the test task in real time. An example is shown in the following figure: + + ``` + submit -m iperf.yaml + ``` + + The following information is displayed on the console: + + ```shell + hi8109@account-vm ~% submit -m iperf.yaml + submit iperf.yaml, got job_id=z9.173923 + query=>{"job_id":["z9.173923"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.173923","message":"","job_state":"submit","result_root":"/srv/result/iperf/2020-11-30/vm-2p8g/openeuler-20.03-aarch6 + {"job_id": "z9.173923", "result_root": "/srv/result/iperf/2020-11-30/vm-2p8g/openeuler-20.03-aarch64/tcp-30/z9.173923", "job_state + {"job_id": "z9.173923", "job_state": "boot"} + {"job_id": "z9.173923", "job_state": "download"} + {"time":"2020-11-30 20:28:16","mac":"0a-f5-9f-83-62-ea","ip":"172.18.192.21","job_id":"z9.173923","state":"running","testbox":"vm- + {"job_state":"running","job_id":"z9.173923"} + {"job_state":"post_run","job_id":"z9.173923"} + {"start_time":"2020-11-30 12:25:15","end_time":"2020-11-30 12:25:45","loadavg":"1.12 0.38 0.14 1/105 1956","job_id":"z9.173923"} + {"job_state":"finished","job_id":"z9.173923"} + {"job_id": "z9.173923", "job_state": "complete"} + {"time":"2020-11-30 20:28:54","mac":"0a-f5-9f-83-62-ea","ip":"172.18.192.21","job_id":"z9.173923","state":"rebooting","testbox":"v + {"job_id": "z9.173923", "job_state": "extract_finished"} + connection closed: normal + ``` + +* **Usage of -c** + + The **-c** parameter must be used together with the **-m** parameter to implement the automatic login function in the task of applying for a device. + + An example is shown in the following figure: + + ``` + submit -m -c borrow-1h.yaml + ``` + + After submitting a task of applying for a device, you will receive the returned login information, such as `ssh ip -p port`. After adding the **-c** parameter, you can log in to the executor without manually entering the SSH login command. + + The following information is displayed on the console: + + ```shell + hi8109@account-vm ~% submit -m -c borrow-1h.yaml + submit borrow-1h.yaml, got job_id=z9.173925 + query=>{"job_id":["z9.173925"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.173925","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-30/vm-2p8g/openeuler-20.03-aarch + {"job_id": "z9.173925", "result_root": "/srv/result/borrow/2020-11-30/vm-2p8g/openeuler-20.03-aarch64/3600/z9.173925", "job_state" + {"job_id": "z9.173925", "job_state": "boot"} + {"job_id": "z9.173925", "job_state": "download"} + {"time":"2020-11-30 20:35:04","mac":"0a-24-5d-c8-aa-d0","ip":"172.18.101.4","job_id":"z9.173925","state":"running","testbox":"vm-2 + {"job_state":"running","job_id":"z9.173925"} + {"job_id": "z9.173925", "state": "set ssh port", "ssh_port": "50200", "tbox_name": "vm-2p8g.taishan200-2280-2s48p-256g--a52-7"} + Host 172.168.131.2 not found in /home/hi8109/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:50200' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Mon Nov 30 12:32:04 CST 2020 + + System load: 0.50 + Processes: 105 + Memory used: 6.1% + Swap used: 0.0% + Usage On: 89% + IP address: 172.17.0.1 + Users online: 1 + + + + root@vm-2p8g ~# + ``` + + You log in to the executor successfully. diff --git a/doc/manual/use-the-compass-ci-platform-to-test-open-source-projects.en.md b/doc/manual/use-the-compass-ci-platform-to-test-open-source-projects.en.md index bb0af45..7a38e5e 100644 --- a/doc/manual/use-the-compass-ci-platform-to-test-open-source-projects.en.md +++ b/doc/manual/use-the-compass-ci-platform-to-test-open-source-projects.en.md @@ -1,71 +1,71 @@ -# How DO I Use the Compass-CI Platform to Test Open Source Projects? - -This document describes how to use the Compass-CI platform to test open source projects. - -### Adding the URL of the Repository to Be Tested to the upstream-repos Repository - -Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: - -1. Fork the code repository to be tested and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. - -![](./../pictures/fork_blacklight.png) - -2. Run the following command to create a file path named with the repository name and its initial letter: - - ``` - mkdir -p b/backlight - ``` - -3. Run the following command to create the **backlight** file in the directory: - - ``` - cd b/backlight - touch backlight - ``` - -4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: - - ``` - vim backlight - ``` - - The format is as follows: - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - > ![](./../public_sys-resources/icon-notice.gif) **Note** - > - > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. - -5. Run the **Pull Request** command to submit the new **backlight** file to the **backlight** repository. - -### Submitting the Test Task to the Compass-CI Platform - -1. Prepare a test case. - - You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) - - * Use the test cases that have been adapted in the repository. - - Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. - - * Compile a test case and add it to the repository. - - For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. - -2. Configure the **auto\_submit.yaml** file and submit the test task. - - You only need to add the following configuration information to the **sbin/auto\_submit.yaml** file in the **compass-ci** repository: - - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - - Submit the modified **auto\_submit.yaml** file to the **compass-ci** repository using Pull Request. Then you can use the Compass-CI platform to test your project. - - For details about how to set parameters in the **auto\_submit.yaml** file, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. \ No newline at end of file +# How DO I Use the Compass-CI Platform to Test Open Source Projects? + +This document describes how to use the Compass-CI platform to test open source projects. + +### Adding the URL of the Repository to Be Tested to the upstream-repos Repository + +Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: + +1. Fork the code repository to be tested and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. + +![](./../pictures/fork_blacklight.png) + +2. Run the following command to create a file path named with the repository name and its initial letter: + + ``` + mkdir -p b/backlight + ``` + +3. Run the following command to create the **backlight** file in the directory: + + ``` + cd b/backlight + touch backlight + ``` + +4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: + + ``` + vim backlight + ``` + + The format is as follows: + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + > ![](./../public_sys-resources/icon-notice.gif) **Note** + > + > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. + +5. Run the **Pull Request** command to submit the new **backlight** file to the **backlight** repository. + +### Submitting the Test Task to the Compass-CI Platform + +1. Prepare a test case. + + You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) + + * Use the test cases that have been adapted in the repository. + + Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. + + * Compile a test case and add it to the repository. + + For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. + +2. Configure the **auto\_submit.yaml** file and submit the test task. + + You only need to add the following configuration information to the **sbin/auto\_submit.yaml** file in the **compass-ci** repository: + + ``` + b/backlight/backlight: + - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml + ``` + + Submit the modified **auto\_submit.yaml** file to the **compass-ci** repository using Pull Request. Then you can use the Compass-CI platform to test your project. + + For details about how to set parameters in the **auto\_submit.yaml** file, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. diff --git a/doc/manual/view-task-results.en.md b/doc/manual/view-task-results.en.md index bd266df..427f74c 100644 --- a/doc/manual/view-task-results.en.md +++ b/doc/manual/view-task-results.en.md @@ -1,23 +1,23 @@ -# How Do I View Task Results? - -After a test case is executed, you can log in to the https://compass-ci.openeuler.org/jobs to view the execution result. - -You can find the task on the website using the suite defined during task submission or the ID returned after the task is submitted, and click the **job\_state** column corresponding to the task to view the task result. - -![](./../pictures/jobs.png) - -## Result Files - -**job.yaml File** - -Some fields in the **job.yaml** file are submitted by users, and other fields in the file are automatically added by the platform based on the submitted jobs. This file contains all the parameters required for the test task. - -**output File** - -The **output** file records the execution process of a test case. The **check\_exit\_code** status code is usually displayed at the end of the file. If the status code is not 0, the test case is incorrect. - -**stats.json** - -After a test case is executed, a file with the same name as that of the test case is generated. The file records the test commands and standardized output results. Compass-CI parses these files and generates a file with the file name extension **.json**. - -The **stats.json** file incorporates all JSON files. The key results of all test commands are included in this file for subsequent comparison and analysis. \ No newline at end of file +# How Do I View Task Results? + +After a test case is executed, you can log in to the https://compass-ci.openeuler.org/jobs to view the execution result. + +You can find the task on the website using the suite defined during task submission or the ID returned after the task is submitted, and click the **job\_state** column corresponding to the task to view the task result. + +![](./../pictures/jobs.png) + +## Result Files + +**job.yaml File** + +Some fields in the **job.yaml** file are submitted by users, and other fields in the file are automatically added by the platform based on the submitted jobs. This file contains all the parameters required for the test task. + +**output File** + +The **output** file records the execution process of a test case. The **check\_exit\_code** status code is usually displayed at the end of the file. If the status code is not 0, the test case is incorrect. + +**stats.json** + +After a test case is executed, a file with the same name as that of the test case is generated. The file records the test commands and standardized output results. Compass-CI parses these files and generates a file with the file name extension **.json**. + +The **stats.json** file incorporates all JSON files. The key results of all test commands are included in this file for subsequent comparison and analysis. diff --git "a/doc/manual/\345\246\202\344\275\225\344\275\277\347\224\250compass-ci\346\265\213\350\257\225\345\274\200\346\272\220\351\241\271\347\233\256.md" "b/doc/manual/\345\246\202\344\275\225\344\275\277\347\224\250compass-ci\346\265\213\350\257\225\345\274\200\346\272\220\351\241\271\347\233\256.md" index e73b9f9..cbbcaef 100644 --- "a/doc/manual/\345\246\202\344\275\225\344\275\277\347\224\250compass-ci\346\265\213\350\257\225\345\274\200\346\272\220\351\241\271\347\233\256.md" +++ "b/doc/manual/\345\246\202\344\275\225\344\275\277\347\224\250compass-ci\346\265\213\350\257\225\345\274\200\346\272\220\351\241\271\347\233\256.md" @@ -1,68 +1,68 @@ -# 使用 compass-ci 平台测试开源项目 - -本文介绍了使用 compass-ci 平台测试开源项目的操作。 - -### 添加待测试仓库 URL 到 upstream-repos 仓库 - -执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 - -1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 - -![](./../pictures/fork_blacklight.png) - -2. 执行以下命令，以首字母和仓库名创建文件路径。 - - ``` - mkdir -p b/backlight - ``` - -3. 执行以下命令，在该目录下新建同名文件 backlight。 - ``` - cd b/backlight - touch backlight - ``` - -4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 - - ``` - vim backlight - ``` - 内容格式为 - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - >![](./../public_sys-resources/icon-notice.gif) **注意：** - > - >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 - -5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 - - -### 提交测试任务到 compass-ci 平台 - -1. 准备测试用例 - - 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 - - * 使用仓库中已经适配好的测试用例 - 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： - iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 - - * 编写测试用例并添加到仓库 - - 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) - -2. 配置 auto_submit.yaml 文件，提交测试任务 - - 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 - - auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 auto_submit.yaml 文件，提交测试任务 + + 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： + ``` + b/backlight/backlight: + - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml + ``` + 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 + + auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 diff --git a/doc/tutorial.md b/doc/tutorial.md index 585be6f..e6f8491 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1,337 +1,337 @@ -# Compass-CI 用户指南 - -## 简介 - - -### 概念 - -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 - - -### 功能描述 - -- 测试服务 - - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 - -- 调测环境登录 - - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 - -- 测试结果分析 - - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 - -- 辅助定位 - - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 - -### 应用场景 - -- 聚合开发者测试用例 - - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 - -- 登录环境随时调测 - - 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 - -- 快照数据分析对比 - - 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 - -- 辅助定界 - - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题 Commit 信息。 - - - -## 使用 Compass-CI -> 当您注册之后，便可以编写 yaml 文件并通过我们的工具上传任务以进行自定义测试，测试功能将尽快上线。 - - -- 注册自己的仓库 - - 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` - -### 编写 job yaml 文件 - -#### job yaml 简介 -job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAML_for_ruby.html)的格式编写，所有 job 文件位于```$LKP_SRC/jobs```路径下。[$LKP_SRC](https://gitee.com/wu_fengguang/lkp-te… - -#### job yaml 的结构 - -- 标识头部 Header（必选） - - 每一个 job 文件的开始部分，都有一些基础的描述信息，称之为 yaml 的 Header。头部主要有 suite 和 category，suite 是测试套的名称。category 是测试类型，包括 benchmarch（性能测试）/functional（功能测试）/noise-benchmark（不常用）。 - - ```yaml - suite: netperf \\测试套名称 - category: benchmark \\测试类型：benchmarch|functional|noise-benchmark - ``` - -- 测试脚本和参数（必选） - - - 测试脚本位于```$LKP_SRC/tests```，```$LKP_SRC/daemon```路径下。job yaml 中写入与上述路径中脚本相匹配的文件，将其视为可执行脚本。 - - ``` - $LKP_SRC/setup - $LKP_SRC/monitors - $LKP_SRC/daemon - $LKP_SRC/tests - ``` - - 参数值作为环境变量传递，每个测试脚本将在其文件头的注释中记录可接受的参数。 - - ```yaml - netserver: - netperf: - runtime: 10s - test: - - TCP_STREAM - - UDP_STREAM - send_size: - - 1024 - - 2048 - ``` -- 测试资源相关变量（必选） - - SUT/schduler/部署等资源相关变量。 - ```yaml - testbox: vm-2p8g - os: openeuler - os_version: 20.03 - os_arch: aarch64 - ``` -- 系统设置脚本（可选） - - 位于```$LKP_SRC/setup```路径下。设置脚本将在测试脚本之前运行，主要是用于启停一些依赖性服务或工具，或者配置测试所需的参数等。 - - ```yaml - cgroup2: #$LKP_SRC/setup/cgroup2 executable script - memory.high: 90% - memory.low: 50% - memory.max: max - memory.swap.max: - io.max: - io.weight: - rdma.max: - ``` -- 监控脚本（可选） - - 位于```$LKP_SRC/monitors```路径下。监视器在运行基准测试时可以捕获性能统计数据，对性能分析和回归根源有价值。 - - ```yaml - proc-vmstats: - nterval: 5 - ``` - -**了解更多 job yaml 相关知识** [Job definition to execution](https://gitee.com/wu_fengguang/lkp-tests/blob/master/jobs/README… - - -### 提交测试任务 - -### 查看测试结果 - -### 比较测试结果 - -### 提交 borrow 任务 - -### 提交 bisect 任务 - -## 高级功能 - -### 添加 OS 支持 - -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 - -#### 操作步骤 - - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs - - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) 加载 docker 镜像 - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) 启动 openEuler 容器 - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) 拷贝 docker 的 rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (openEuler为例) - - - 1) 下载 openEule r官方网站提供的 qcow2 格式镜像 - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) 使用{compass-ci}/container/qcow2rootfs 制作rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. 定制rootfs - 1. 使用chroot命令切换到 rootfs (此步骤需要 root 权限) - ```bash - chroot openEuler-rootfs - ``` - 2. 根据个人需要安装并配置服务 - - a. 修改 root 密码 - b. 配置 ssh 服务 - c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： - > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核rpm包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 - - 3. 退出 rootfs，并打包 - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. 日志报错 “Unable to mount root fs on unknown-block” - - 问题现象 - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - 解决方法 - - 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 - -2. 系统运行缓慢 - - - 问题现象 - - 打包镜像体积过大，会消耗很大内存 - - 解决方法 - - 建议用户根据具体需要对 rootfs 进行裁剪。 - - -### 添加测试用例 - 参考文档详见：[添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/ad… - - -### PKGBUILD 构建 - -1. 概述： - - 使用 PKGBUILD 完成上游项目源代码的编译构建，同一项目可以使用不同 PKGBUILD 完成项目编译构建过程。 - PKGBUILD 相关参考文档：https://wiki.archlinux.org/index.php/PKGBUILD。 - -2. 步骤： - - 1) 注册仓库 - 如果您想在 git push 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` - - 2) 执行构建测试 - 项目代码更新后自动触发构建任务，无需其他操作。 - - 3) 查看结果 - web: https://compass-ci.openeuler.org/jobs。 - - 4) 备注 - 目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 - PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 - - -### depends - -- 依赖包构建 - 流程: 以下默认使用 debian:sid 版本进行,如果使用其他OS跑测试用例，则需要在 lkp-tests/distro/adaptation/ 下将包名以键值对的形式写入对应 OS 的文件中 - > 例： `echo 'apache2: httpd' >> lkp-tests/distro/adaptation/openeuler` - - 1) 在lkp-tests/distro/depends/下新建一个与测试用例同名的文件 - `vim lkp-tests/distro/depends/test` - 2) 将测试用例所需debian下的包名填入创建的文件中 - `echo "apache2" >> lkp-tests/distro/depends/test` - 3) 生成lkp-aarch64.cgz - `./crystal-ci/container/lkp-initrd/run` - 4) 提交测试用例 - `submit test.yaml` - 5) 查看结果 - https://compass-ci.openeuler.org/jobs - - - -### 本地搭建 compass-ci 服务器节点 - -在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 - -#### 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) - - >**说明：** - >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - - >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… - - -#### 操作指导 -1. 登录 openEuler 系统 - -2. 创建工作目录并设置文件权限 - ```bash - mkdir demo && cd demo && umask 002 - ``` -3. 克隆 compass-ci 项目代码到 demo 目录 - ```bash - git clone https://gitee.com/wu_fengguang/compass-ci.git - ``` -4. 执行一键部署脚本 install-tiny - ```bash - cd compass-ci/sparrow && ./install-tiny - ``` - - -## 呼吁合作 - - 增强 git bisect 能力 - - 增强数据分析能力 - - 增强数据结果可视化能力 - -## 下一步计划 - -1. 优化 git bisect 精准率及效率。 -2. 优化数据分析联动能力，让数据分析更加聚焦。 -3. 数据可视化优化，更加友好展示数据比较结果。 -4. 增强登录认证机制，如：GPG 认证。 -5. 优化部署效率。 +# Compass-CI 用户指南 + +## 简介 + + +### 概念 + +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 + + +### 功能描述 + +- 测试服务 + + 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + +- 调测环境登录 + + Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + +- 测试结果分析 + + Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 + +- 辅助定位 + + Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 + +### 应用场景 + +- 聚合开发者测试用例 + + 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 + +- 登录环境随时调测 + + 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 + +- 快照数据分析对比 + + 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 + +- 辅助定界 + + 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题 Commit 信息。 + + + +## 使用 Compass-CI +> 当您注册之后，便可以编写 yaml 文件并通过我们的工具上传任务以进行自定义测试，测试功能将尽快上线。 + + +- 注册自己的仓库 + + 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + ```bash + git clone https://gitee.com/wu_fengguang/upstream-repos.git + less upstream-repos/README.md + ``` + +### 编写 job yaml 文件 + +#### job yaml 简介 +job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAML_for_ruby.html)的格式编写，所有 job 文件位于```$LKP_SRC/jobs```路径下。[$LKP_SRC](https://gitee.com/wu_fengguang/lkp-te… + +#### job yaml 的结构 + +- 标识头部 Header（必选） + + 每一个 job 文件的开始部分，都有一些基础的描述信息，称之为 yaml 的 Header。头部主要有 suite 和 category，suite 是测试套的名称。category 是测试类型，包括 benchmarch（性能测试）/functional（功能测试）/noise-benchmark（不常用）。 + + ```yaml + suite: netperf \\测试套名称 + category: benchmark \\测试类型：benchmarch|functional|noise-benchmark + ``` + +- 测试脚本和参数（必选） + + - 测试脚本位于```$LKP_SRC/tests```，```$LKP_SRC/daemon```路径下。job yaml 中写入与上述路径中脚本相匹配的文件，将其视为可执行脚本。 + + ``` + $LKP_SRC/setup + $LKP_SRC/monitors + $LKP_SRC/daemon + $LKP_SRC/tests + ``` + - 参数值作为环境变量传递，每个测试脚本将在其文件头的注释中记录可接受的参数。 + + ```yaml + netserver: + netperf: + runtime: 10s + test: + - TCP_STREAM + - UDP_STREAM + send_size: + - 1024 + - 2048 + ``` +- 测试资源相关变量（必选） + + SUT/schduler/部署等资源相关变量。 + ```yaml + testbox: vm-2p8g + os: openeuler + os_version: 20.03 + os_arch: aarch64 + ``` +- 系统设置脚本（可选） + + 位于```$LKP_SRC/setup```路径下。设置脚本将在测试脚本之前运行，主要是用于启停一些依赖性服务或工具，或者配置测试所需的参数等。 + + ```yaml + cgroup2: #$LKP_SRC/setup/cgroup2 executable script + memory.high: 90% + memory.low: 50% + memory.max: max + memory.swap.max: + io.max: + io.weight: + rdma.max: + ``` +- 监控脚本（可选） + + 位于```$LKP_SRC/monitors```路径下。监视器在运行基准测试时可以捕获性能统计数据，对性能分析和回归根源有价值。 + + ```yaml + proc-vmstats: + nterval: 5 + ``` + +**了解更多 job yaml 相关知识** [Job definition to execution](https://gitee.com/wu_fengguang/lkp-tests/blob/master/jobs/README… + + +### 提交测试任务 + +### 查看测试结果 + +### 比较测试结果 + +### 提交 borrow 任务 + +### 提交 bisect 任务 + +## 高级功能 + +### 添加 OS 支持 + +制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 + +#### 操作步骤 + + 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） + - 通过 docker 获取 rootfs + + 1) 下载 openEuler 官方提供的 docker 镜像压缩包 + + ```bash + wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… + ``` + 2) 加载 docker 镜像 + ```bash + docker load -i openEuler-docker.aarch64 + ``` + + 3) 启动 openEuler 容器 + ```bash + docker run -id openeuler-20.03-lts + ``` + 4) 拷贝 docker 的 rootfs + ```bash + docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs + ``` + - 通过 qemu.img(qcow2格式)获取 rootfs (openEuler为例) + + + 1) 下载 openEule r官方网站提供的 qcow2 格式镜像 + ```bash + wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… + ``` + 2) 使用{compass-ci}/container/qcow2rootfs 制作rootfs + ```bash + cd {compass-ci}/container/qcow2rootfs + ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs + ``` + 2. 定制rootfs + 1. 使用chroot命令切换到 rootfs (此步骤需要 root 权限) + ```bash + chroot openEuler-rootfs + ``` + 2. 根据个人需要安装并配置服务 + + a. 修改 root 密码 + b. 配置 ssh 服务 + c. 检查系统时间 + d. 如果使用 docker 制作 osimage 还需要以下操作： + > 1. 安装所需版本内核 + > 2. 从 centos 官方网站下载内核rpm包 + > 3. 使用 yum 进行安装 + > 4. 删除 docker 环境变量文件 + + 3. 退出 rootfs，并打包 + ```bash + cd $rootfs + find . | coip -o -Hnewc |gzip -9 > $os_name.cgz + ``` +#### FAQ +1. 日志报错 “Unable to mount root fs on unknown-block” + - 问题现象 + ```bash + [ 0.390437] List of all partitions: + [ 0.390806] No filesystem could mount root, tried: + [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) + ... + [ 0.399404] Memory Limit: none + [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- + ``` + - 解决方法 + + 1）启动内存不足，增加内存可解决。 + 2）内核文件权限不足，给予 644 权限。 + +2. 系统运行缓慢 + + - 问题现象 + + 打包镜像体积过大，会消耗很大内存 + - 解决方法 + + 建议用户根据具体需要对 rootfs 进行裁剪。 + + +### 添加测试用例 + 参考文档详见：[添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/ad… + + +### PKGBUILD 构建 + +1. 概述： + + 使用 PKGBUILD 完成上游项目源代码的编译构建，同一项目可以使用不同 PKGBUILD 完成项目编译构建过程。 + PKGBUILD 相关参考文档：https://wiki.archlinux.org/index.php/PKGBUILD。 + +2. 步骤： + + 1) 注册仓库 + 如果您想在 git push 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + ```bash + git clone https://gitee.com/wu_fengguang/upstream-repos.git + less upstream-repos/README.md + ``` + + 2) 执行构建测试 + 项目代码更新后自动触发构建任务，无需其他操作。 + + 3) 查看结果 + web: https://compass-ci.openeuler.org/jobs。 + + 4) 备注 + 目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 + PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 + + +### depends + +- 依赖包构建 + 流程: 以下默认使用 debian:sid 版本进行,如果使用其他OS跑测试用例，则需要在 lkp-tests/distro/adaptation/ 下将包名以键值对的形式写入对应 OS 的文件中 + > 例： `echo 'apache2: httpd' >> lkp-tests/distro/adaptation/openeuler` + + 1) 在lkp-tests/distro/depends/下新建一个与测试用例同名的文件 + `vim lkp-tests/distro/depends/test` + 2) 将测试用例所需debian下的包名填入创建的文件中 + `echo "apache2" >> lkp-tests/distro/depends/test` + 3) 生成lkp-aarch64.cgz + `./crystal-ci/container/lkp-initrd/run` + 4) 提交测试用例 + `submit test.yaml` + 5) 查看结果 + https://compass-ci.openeuler.org/jobs + + + +### 本地搭建 compass-ci 服务器节点 + +在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 + +#### 准备工作 +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + + >**说明：** + >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + + +#### 操作指导 +1. 登录 openEuler 系统 + +2. 创建工作目录并设置文件权限 + ```bash + mkdir demo && cd demo && umask 002 + ``` +3. 克隆 compass-ci 项目代码到 demo 目录 + ```bash + git clone https://gitee.com/wu_fengguang/compass-ci.git + ``` +4. 执行一键部署脚本 install-tiny + ```bash + cd compass-ci/sparrow && ./install-tiny + ``` + + +## 呼吁合作 + - 增强 git bisect 能力 + - 增强数据分析能力 + - 增强数据结果可视化能力 + +## 下一步计划 + +1. 优化 git bisect 精准率及效率。 +2. 优化数据分析联动能力，让数据分析更加聚焦。 +3. 数据可视化优化，更加友好展示数据比较结果。 +4. 增强登录认证机制，如：GPG 认证。 +5. 优化部署效率。 -- 2.23.0

1 0

[PATCH compass-ci 09/40] doc/manual: fix PKGBUILD doc
by Wu Zhende 15 Dec '20

15 Dec '20

From: Wu Fengguang <wfg(a)mail.ustc.edu.cn> Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- ...le-pkgbuild.en.md => write-pkgbuild.en.md} | 164 +++++++++--------- 1 file changed, 79 insertions(+), 85 deletions(-) rename doc/manual/{compile-pkgbuild.en.md => write-pkgbuild.en.md} (61%) diff --git a/doc/manual/compile-pkgbuild.en.md b/doc/manual/write-pkgbuild.en.md similarity index 61% rename from doc/manual/compile-pkgbuild.en.md rename to doc/manual/write-pkgbuild.en.md index 097e0b8..7969b71 100644 --- a/doc/manual/compile-pkgbuild.en.md +++ b/doc/manual/write-pkgbuild.en.md @@ -1,85 +1,79 @@ -# PKGBUILD - -A PKGBUILD is a shell script. makepkg uses the instructions contained in PKGBUILD to generate a software package that incorporates binary files and installation instructions. - -# What Does PKGBUILD Include? - -PKGBUILD includes variables and functions. - -## Defining a Variable - -- pkgname: Mandatory. It indicates the name of a software package. -- pkgver: Mandatory. It indicates the version of a software package. -- pkgrel: Mandatory. It indicates the release number of a software package. -- arch: Mandatory. It indicates the architecture sequence of a software package. -- depends: Optional. It indicates the name of the dependency package required for running a software test. -- makedepends: Optional. It indicates the list of files required for building a software package. -- source: Optional. It indicates the MD5 hash value of each source file to verify the integrity of the source file during the build process. - -## Defining a Function - -- package function - - Mandatory. It is used for installing files to the directory that will be the root directory of the build package. - -- prepare function - - Optional. It is used for executing the operation of building source code. - -- build function - - Optional. It is used for compiling and/or building source code. - -- check function - - Optional. It is used for running the test suite of the program package. - -> ![](./../public_sys-resources/icon-notice.gif) **Notice** -> -> **srcdir** is the directory for extracting or copying source files. All packaging functions run in the **srcdir** directory. **pkgdir** is the root directory for building software packages and is used only in the package function. - -# How Do I Compile PKGBUILD? - -1. Run the following command to create a PKGBUILD. The file name must be **PKGBUILD**. - - ```shell - touch PKGBUILD - ``` - -2. Use the vim editor to open the **PKGBUILD** file and compile it. The following is an example of the **PKGBUILD** file: - - ```shell - pkgname=zstd - pkgver=1.4.4 - pkgrel=2 - arch=('i686' 'x86_64' 'aarch64') - url='https://github.com/facebook/zstd' - license=('custom:BSD3' 'GPL2') - depends=('xz' 'zlib' 'lz4') - makedepends=('git') - source=('git://github.com/facebook/zstd.git#branch=dev') - md5sums=('SKIP') - - pkgver() { - cd "$srcdir/$pkgname" - git describe --long --tags | sed 's/$[^-]*-g$/r\1/;s/-/./g;s/^v//g' - } - - build() { - cd "$srcdir/$pkgname" - make - make -C contrib/pzstd - } - - package() { - cd "$srcdir/$pkgname" - make PREFIX="/usr" DESTDIR="$pkgdir/" install - install -D -m755 contrib/pzstd/pzstd "$pkgdir/usr/bin/pzstd" - install -D -m644 LICENSE "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE" - } - ``` - -# Reference - -- [PKGBUILD(5) Manual Page](https://www.archlinux.org/pacman/PKGBUILD.5.html) -- [pkgbuild demo](https://git.archlinux.org/pacman.git/plain/proto/PKGBUILD.proto) \ No newline at end of file +# PKGBUILD + +A PKGBUILD is a shell script. makepkg uses the instructions contained in PKGBUILD to generate a software package that incorporates binary files and installation instructions. + +# What Does PKGBUILD Include? + +PKGBUILD includes variables and functions. + +## Defining a Variable + +- pkgname: Mandatory. It indicates the name of a software package. +- pkgver: Mandatory. It indicates the version of a software package. +- pkgrel: Mandatory. It indicates the release number of a software package. +- arch: Mandatory. It indicates the architecture sequence of a software package. +- depends: Optional. It indicates the name of the dependency package required for running a software test. +- makedepends: Optional. It indicates the list of files required for building a software package. +- source: Optional. It indicates the MD5 hash value of each source file to verify the integrity of the source file during the build process. + +## Defining a Function + +- package function + + Mandatory. It is used for installing files to the directory that will be the root directory of the build package. + +- prepare function + + Optional. It is used for executing the operation of building source code. + +- build function + + Optional. It is used for compiling and/or building source code. + +- check function + + Optional. It is used for running the test suite of the program package. + +> ![](./../public_sys-resources/icon-notice.gif) **Notice** +> +> **srcdir** is the directory for extracting or copying source files. All packaging functions run in the **srcdir** directory. **pkgdir** is the root directory for building software packages and is used only in the package function. + +# How do i write PKGBUILD? + +1. Read official documents: + +- [PKGBUILD(5) Manual Page](https://www.archlinux.org/pacman/PKGBUILD.5.html) +- [pkgbuild demo](https://git.archlinux.org/pacman.git/plain/proto/PKGBUILD.proto) + +2. Create **PKGBUILD** file with vim or other editor. The following is an example of the **PKGBUILD** file: + +```shell +pkgname=zstd +pkgver=1.4.4 +pkgrel=2 +arch=('i686' 'x86_64' 'aarch64') +url='https://github.com/facebook/zstd' +license=('custom:BSD3' 'GPL2') +depends=('xz' 'zlib' 'lz4') +makedepends=('git') +source=('git://github.com/facebook/zstd.git#branch=dev') +md5sums=('SKIP') + +pkgver() { + cd "$srcdir/$pkgname" + git describe --long --tags | sed 's/$[^-]*-g$/r\1/;s/-/./g;s/^v//g' +} + +build() { + cd "$srcdir/$pkgname" + make + make -C contrib/pzstd +} + +package() { + cd "$srcdir/$pkgname" + make PREFIX="/usr" DESTDIR="$pkgdir/" install + install -D -m755 contrib/pzstd/pzstd "$pkgdir/usr/bin/pzstd" + install -D -m644 LICENSE "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE" +} +``` -- 2.23.0

1 0

[PATCH compass-ci 08/40] doc/manual: rename English manual files
by Wu Zhende 15 Dec '20

15 Dec '20

From: Wu Fengguang <wfg(a)mail.ustc.edu.cn> Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- ...apply-for-a-test-machine.md => apply-for-a-test-machine.en.md} | 0 ...ow-do-i-compare-test-results.md => compare-test-results.en.md} | 0 .../{how-do-i-compile-pkgbuild.md => compile-pkgbuild.en.md} | 0 ...se-the-compass-ci-platform-to-test-open-source-projects.en.md} | 0 .../{how-do-i-view-task-results.md => view-task-results.en.md} | 0 5 files changed, 0 insertions(+), 0 deletions(-) rename doc/manual/{how-do-i-apply-for-a-test-machine.md => apply-for-a-test-machine.en.md} (100%) rename doc/manual/{how-do-i-compare-test-results.md => compare-test-results.en.md} (100%) rename doc/manual/{how-do-i-compile-pkgbuild.md => compile-pkgbuild.en.md} (100%) rename doc/manual/{how-do-i-use-the-compass-ci-platform-to-test-open-source-projects.md => use-the-compass-ci-platform-to-test-open-source-projects.en.md} (100%) rename doc/manual/{how-do-i-view-task-results.md => view-task-results.en.md} (100%) diff --git a/doc/manual/how-do-i-apply-for-a-test-machine.md b/doc/manual/apply-for-a-test-machine.en.md similarity index 100% rename from doc/manual/how-do-i-apply-for-a-test-machine.md rename to doc/manual/apply-for-a-test-machine.en.md diff --git a/doc/manual/how-do-i-compare-test-results.md b/doc/manual/compare-test-results.en.md similarity index 100% rename from doc/manual/how-do-i-compare-test-results.md rename to doc/manual/compare-test-results.en.md diff --git a/doc/manual/how-do-i-compile-pkgbuild.md b/doc/manual/compile-pkgbuild.en.md similarity index 100% rename from doc/manual/how-do-i-compile-pkgbuild.md rename to doc/manual/compile-pkgbuild.en.md diff --git a/doc/manual/how-do-i-use-the-compass-ci-platform-to-test-open-source-projects.md b/doc/manual/use-the-compass-ci-platform-to-test-open-source-projects.en.md similarity index 100% rename from doc/manual/how-do-i-use-the-compass-ci-platform-to-test-open-source-projects.md rename to doc/manual/use-the-compass-ci-platform-to-test-open-source-projects.en.md diff --git a/doc/manual/how-do-i-view-task-results.md b/doc/manual/view-task-results.en.md similarity index 100% rename from doc/manual/how-do-i-view-task-results.md rename to doc/manual/view-task-results.en.md -- 2.23.0

1 0

[PATCH compass-ci 07/40] container: remains format consistent
by Wu Zhende 15 Dec '20

15 Dec '20

From: Liu Yinsi <liuyinsi(a)163.com> keep start script content format consistent with others, looks more suitable. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/git-daemon/start | 12 +++++++++++- container/redis/start | 13 ++++++++++++- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/container/git-daemon/start b/container/git-daemon/start index 5b4c90b..72330aa 100755 --- a/container/git-daemon/start +++ b/container/git-daemon/start @@ -6,8 +6,18 @@ docker_rm git-daemon -docker run --restart=always -d -p 9418:9418 -v /srv/git:/git -v /etc/localtime:/etc/localtime:ro --name git-daemon apline311:git-daemon +cmd=( + docker run + --restart=always + --name git-daemon + -d + -p 9418:9418 + -v /srv/git:/git + -v /etc/localtime:/etc/localtime:ro + apline311:git-daemon +) +"${cmd[@]}" # test diff --git a/container/redis/start b/container/redis/start index 827ad0b..34266a0 100755 --- a/container/redis/start +++ b/container/redis/start @@ -10,7 +10,18 @@ docker_rm redis507s01 -docker run --restart=always -d -p 6379:6379 -v /srv/redis/alpine/server01:/srv/redis -v /etc/localtime:/etc/localtime:ro --name redis507s01 redis507n:alpine311 +cmd=( + docker run + --restart=always + --name redis507s01 + -d + -p 6379:6379 + -v /srv/redis/alpine/server01:/srv/redis + -v /etc/localtime:/etc/localtime:ro + redis507n:alpine311 +) + +"${cmd[@]}" # test echo you can use cmd: redis-cli to start a redis client -- 2.23.0

1 0

[PATCH compass-ci 06/40] keep curl command silent mode and show error
by Wu Zhende 15 Dec '20

15 Dec '20

From: Liu Yinsi <liuyinsi(a)163.com> change that to '-sS' will silent output but the error info Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/start | 2 +- sbin/es-accounts-mapping.sh | 4 ++-- sbin/es-jobs-mapping.sh | 6 +++--- sbin/es-regression-mapping.sh | 4 ++-- sbin/es-repo-mapping.sh | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/container/logging-es/start b/container/logging-es/start index 25ed36b..66464e6 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -23,7 +23,7 @@ cmd=( echo "Please wait 60s for setting index" sleep 60 -curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' { "index": { "refresh_interval": "120s", diff --git a/sbin/es-accounts-mapping.sh b/sbin/es-accounts-mapping.sh index 635f4fc..d5566dc 100755 --- a/sbin/es-accounts-mapping.sh +++ b/sbin/es-accounts-mapping.sh @@ -3,14 +3,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # check whether accounts index has created -status_code=$(curl -sIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/accounts) +status_code=$(curl -sSIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/accounts) if [ "$status_code" -eq 200 ] then echo "accounts index has been created, exit." else echo "begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -d '{ "mappings": { "_doc": { "dynamic": false, diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 52517b9..5cb9b9c 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -28,14 +28,14 @@ then fi # Determine whether jobs index has created -status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/jobs) +status_code=$(curl -sSIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/jobs) if [ $status_code -eq 200 ] then echo "jobs index has create, exit." else echo "jobs index not exists, begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ "mappings": { "_doc": { "dynamic": false, @@ -164,7 +164,7 @@ else echo "create jobs index failed." else echo "set index.mapping.total_fields.limit: 10000" - curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ + curl -sS -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ -d '{"index.mapping.total_fields.limit": 10000}' fi fi diff --git a/sbin/es-regression-mapping.sh b/sbin/es-regression-mapping.sh index 7ef19cd..8e04bdb 100755 --- a/sbin/es-regression-mapping.sh +++ b/sbin/es-regression-mapping.sh @@ -1,14 +1,14 @@ #!/bin/sh # check whether regression index has created -status_code=$(curl -sIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/regression) +status_code=$(curl -sSIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/regression) if [ "$status_code" -eq 200 ] then echo "regression index has been created, exit." else echo "begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -d '{ "mappings": { "_doc": { "dynamic": false, diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh index 4605051..17f0986 100755 --- a/sbin/es-repo-mapping.sh +++ b/sbin/es-repo-mapping.sh @@ -3,12 +3,12 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # Check if "repo" index already exists. -status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +status_code=$(curl -sSIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) [ $status_code -eq 200 ] && echo '"repo" index already exists.' && exit # Create "repo" index. echo 'Start to create "repo" index.' -curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' { "mappings": { "_doc": { -- 2.23.0

1 0

[PATCH compass-ci 05/40] lib/*: fix spell error
by Wu Zhende 15 Dec '20

15 Dec '20

From: Wang Yong <wangyong0117(a)qq.com> mutli -> multi Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/es_query.rb | 5 ++--- src/lib/web_backend.rb | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index a250eaf..68c70b5 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -23,7 +23,7 @@ class ESQuery warn 'empty filter!' exit end - query_fields = build_mutli_field_subquery_body items + query_fields = build_multi_field_subquery_body items query = { query: { bool: { @@ -72,8 +72,7 @@ end # } # } # items['range'] = range -# build_mutli_field_subquery_body(items) -def build_mutli_field_subquery_body(items) +def build_multi_field_subquery_body(items) query_fields = [] items.each do |key, value| if value.is_a?(Array) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 014600a..94883a3 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -163,7 +163,7 @@ def do_get_groups_matrices(query, dimension, total, size, from) end def get_groups_matrices(conditions, dimension, must, size, from) - must += build_mutli_field_subquery_body(conditions) + must += build_multi_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } total = es_count(count_query) return {} if total < 1 -- 2.23.0

1 0

[PATCH compass-ci 04/40] container/srv-http: turn off statistic folder size function
by Wu Zhende 15 Dec '20

15 Dec '20

From: Lu Weitao <luweitaobe(a)163.com> [Why] When the folder is larger, the web page response is slower Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/srv/_h5ai/private/conf/options.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/srv-http/root/srv/_h5ai/private/conf/options.json b/container/srv-http/root/srv/_h5ai/private/conf/options.json index 69a7add..5d922da 100755 --- a/container/srv-http/root/srv/_h5ai/private/conf/options.json +++ b/container/srv-http/root/srv/_h5ai/private/conf/options.json @@ -158,7 +158,7 @@ - type: string, "php" (sloooow) or "shell-du" (sloow) */ "foldersize": { - "enabled": true, + "enabled": false, "type": "php" }, -- 2.23.0

1 0

[PATCH compass-ci 03/40] container/srv-http: fix can not display image
by Wu Zhende 15 Dec '20

15 Dec '20

From: Lu Weitao <luweitaobe(a)163.com> [Why] Haven't define images relationship in mime.types Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/mime.types | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index a4ee74b..a8f7b0c 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -6,6 +6,17 @@ types { text/yaml yaml; text/plain txt time data sh log; + image/gif gif; + image/jpeg jpeg jpg; + image/png png; + image/svg+xml svg svgz; + image/tiff tif tiff; + image/vnd.wap.wbmp wbmp; + image/webp webp; + image/x-icon ico; + image/x-jng jng; + image/x-ms-bmp bmp; + application/json json; application/x-gzip gz tgz cgz; } -- 2.23.0

1 0

[PATCH compass-ci 02/40] container/srv-http: config php-fpm user
by Wu Zhende 15 Dec '20

15 Dec '20

From: Lu Weitao <luweitaobe(a)163.com> Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 2821e57..8f9a36d 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -7,5 +7,9 @@ ADD root / RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf +RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf +RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf + +RUN chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] -- 2.23.0

1 0