January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] providers/libvirt: add a libvirt provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

request api: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 67 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100755 providers/libvirt/run.rb diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb new file mode 100755 index 0000000..0ff0d4f --- /dev/null +++ b/providers/libvirt/run.rb @@ -0,0 +1,67 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'digest/md5' +require_relative "#{ENV['CCI_SRC']}/lib/log" +require_relative "#{ENV['CCI_SRC']}/lib/sched_client" +require_relative "#{ENV['CCI_SRC']}/providers/lib/context" + +def create_logger(hostname) + filename = "#{hostname}.log" + File.delete(filename) if FileTest.exist?(filename) + Log.new filename +end + +def compute_mac(string) + /(..)(..)(..)(..)(..)/ =~ Digest::MD5.hexdigest(string) + "0a:#{$1}:#{$2}:#{$3}:#{$4}:#{$5}" +end + +def nojob?(response) + flag = false + if response['job_id'].empty? + puts '----------' + puts 'No job now' + puts '----------' + flag = true + end + return flag +end + +def request(context, sched_client, logger) + mac = context.info['mac'] + hostname = context.info['hostname'] + queues = context.info['queues'] + sched_client.register_mac2host(hostname, mac) + sched_client.register_host2queues(hostname, queues) + response = JSON.parse(sched_client.consume_job('libvirt', 'mac', mac)) + if nojob?(response) + logger.info('No job now') + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) + response = nil + end + return response +end + +def main(hostname, queues) + logger = create_logger(hostname) + mac = compute_mac hostname + context = Context.new(mac, hostname, queues) + sched_client = SchedClient.new + response = request(context, sched_client, logger) + return if response.nil? + + # TODO + # based on response load file + # TODO + # generate domain.xml + # TODO + # use ruby-libvirt api start a vm + # clean env + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) +end -- 2.23.0

3 4

[PATCH lkp-tests] tests/build-pkg: export V=1 to print verbose output
by Lin Jiaxin 26 Jan '21

26 Jan '21

A developer who really wants to make silent rules enabled by default will calling AM_SILENT_RULES([yes]) in configure.ac. It make compiling packages by default does not show the full compilation/link command lines, but only short "summaries", like this: EPOINFO main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/ctags-cmd.o make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/libctags_a-args.o However, we need the full compilation/link command lines, like this: ./misc/gen-repoinfo main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' gcc -DHAVE_CONFIG_H -I. -I. -I. -I./main -I./peg -DHAVE_PACKCC -DUSE_SYSTEM_STRNLEN -DHAVE_REPOINFO_H -D_FORTIFY_SOURCE=2 -std=gnu99 -Wall -I/usr/include/libxml2 -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -MT main/ctags-cmd.o -MD -MP -MF main/.deps/ctags-cmd.Tpo -c -o main/ctags-cmd.o `test -f 'main/cmd.c' || echo './'`main/cmd.c make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' mv -f main/.deps/ctags-cmd.Tpo main/.deps/ctags-cmd.Po make V=1 will produce verbose output, so export V=1 in build-pkg script. refer to: https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rule… https://lists.gnu.org/archive/html/bug-autoconf/2012-01/msg00009.html Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 5344da2a..f3545a19 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -157,6 +157,8 @@ build_source_pkg() local mark="true" [ "${os}" == "archlinux" ] && local mark="pacman" + export V=1 + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf -- 2.23.0

2 1

[PATCH compass-ci] doc/manual: optimize borrow-machine.en.md
by Luan Shengde 26 Jan '21

26 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/borrow-machine.en.md | 341 +++++++++----------------------- 1 file changed, 98 insertions(+), 243 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index 6a1fe19..36823c3 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -1,246 +1,101 @@ -# Prerequisites - -Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - -- Send an email to apply for an account. -- Receive an email from compass-ci-robot(a)qq.com. -- Configure the local environment. - -# Applying for a Test Machine (VM) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (VM). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170593 - query=>{"job_id":["z9.170593"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} - {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} - {"job_id": "z9.170593", "job_state": "boot"} - {"job_id": "z9.170593", "job_state": "download"} - {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - {"job_state":"running","job_id":"z9.170593"} - {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 06:44:18 CST 2020 - - System load: 0.83 - Processes: 107 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.18.156.13 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (VM) after use. - - ```shell - root@vm-2p8g ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - -# Applying for a Test Machine (Physical Machine) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (physical machine). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170594 - query=>{"job_id":["z9.170594"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} - {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} - {"job_id": "z9.170594", "job_state": "boot"} - {"job_id": "z9.170594", "job_state": "download"} - {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} - {"job_state":"running","job_id":"z9.170594"} - {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 14:51:59 CST 2020 - - System load: 1.31 - Processes: 1020 - Memory used: 5.1% - Swap used: 0.0% - Usage On: 3% - IP address: 172.168.178.48 - Users online: 1 - - - - root@taishan200-2280-2s64p-256g--a5 ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (physical machine) after use. - - ```shell - root@taishan200-2280-2s64p-256g--a5 ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` +# Prepare + +- Apply account +- Config default yaml files + +If have not completed above works, reference to [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… to finish it. + +# Apply testbox + +## 1. Generate local ssh pub key + + Use the command below to check the pub key exists: + + ls ~/.ssh/*.pub + + Generate one if you don't have one now: + + ssh-keygen + +## 2. Select the job yaml + + We have provided various examples for you in dir ~/lkp-tests/jobs: + + You can filtrate examples for borrowing machine with command below: + + cd ~/lkp-tests/jobs + ls borrow* + +## 3. Submit job + + Command to submit jobs: + + for DCs: + + submit -c -m testbox=dc-2g os_mount=container docker_image=centos:8 borrow-1h.yaml + + for VMs: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + for HWs: + + submit -c -m testbox=taishan200-2280-2s48p-256g borrow-1h.yaml + + - You can view the logs in real time for the job with the command above。 + - You will directly login the testbox if the job runs successfully. + - And you will receive an email that contains login command and server configuration information. + - Only within the period of borrowing machine, you can access the textbox with the login command. + +## 4. Return testbox + + Reruan manually(recommended): + + Manually execute 'reboot' in time to return the testbox. + Avoid a waste of computer resource with no-load running. + + Automatically return on maturity: + + The testbox will be returned automatically if it has expired its service life. + + - All testboxes will be returned if it has been executed command 'reboot'. + - The testbox cannot be accessed any more after it has been returned. + - Apply a new one if you want continue to use it. # FAQ -* How Do I Change the Duration of Keeping the Test Machine when Applying for It? - - ```shell - hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml - suite: borrow - testcase: borrow - - runtime: 1h - ssh_pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: - hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the runtime field. - hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml - # After changing the value, submit the request again. - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - ``` - -* Guide to the **submit** Command - - Reference: [submit-job.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… - -* What Are the testbox Options? - - For details about the testbox options, visit https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts. - - > ![](./../public_sys-resources/icon-note.gif) **Note** - > - > VM testbox: vm-xxx - > - > PM testbox: taishan200-2280-xxx - - > ![](./../public_sys-resources/icon-notice.gif) **Notice** - > -> - If the testbox of a physical machine ends with `--axx`, a physical machine is specified. If a task is already in the task queue of the physical machine, the borrow task you submitted will not be processed until the previous task in the queue is completed. - > - If the testbox of a physical machine does not end with `-axx`, no physical machine is specified. In this case, the borrow task you submitted will be immediately allocated to idle physical machines in the cluster for execution. - -* How Do I Borrow the Specified Operating System? - - For details about the supported `os`, `os_arch`, and `os_version`, see [os-os\_verison-os\_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/…. +## Customize the borrowing time + + Find key in the yaml file and edit its value according to your requirement. + + - The borrowing period can be calculated in days and hours. + - The maximum period is no more than 10 day. + +## Guidance for command submit + + See the usage and options for command 'submit' the command below: + + submit -h + + Reference the following line to learn the advanced usage for command 'submit': + + [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… + +## Available testbox + + For a full list of testbox, reference to https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + + >![](./../public_sys-resources/icon-note.gif) **instruction:** + > + > - DCs: dc-xxx + > - VMs: vm-xxx + > - HWs: taishan200-2280-xxx + + >![](./../public_sys-resources/icon-notice.gif) **attention:** + > - It means that you choosed a specified physical machine if the testbox name is end with `--axx`. + > - You will need to wait if there are already tasks in the task queue for the machine. + > - Your job will be randomly assigned to a machine that meets the requirements if the testbox name is not end with '-axx'. + +## Specify the OS + + About supportted `os`, `os_arch`, `os_version`, reference to [os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/ma… -- 2.23.0

1 0

[PATCH lkp-tests] lib/log.sh: add square bracket
by Liu Yinsi 26 Jan '21

26 Jan '21

before: 2021-01-26 14:53:36 ERROR -- cifs module is not loaded in after: 2021-01-26 15:00:27 [ERROR] -- cifs module is not loaded in [why] use square bracket make log more obvious. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/log.sh b/lib/log.sh index 5acf61825..207795ece 100644 --- a/lib/log.sh +++ b/lib/log.sh @@ -2,25 +2,25 @@ log() { - echo "$(date +'%F %T') INFO -- $*" + echo "$(date +'%F %T') [INFO] -- $*" } log_info() { - echo "$(date +'%F %T') INFO -- $*" + echo "$(date +'%F %T') [INFO] -- $*" } log_debug() { - echo "$(date +'%F %T') DEBUG -- $*" + echo "$(date +'%F %T') [DEBUG] -- $*" } log_warn() { - echo "$(date +'%F %T') WARN -- $*" >&2 + echo "$(date +'%F %T') [WARN] -- $*" >&2 } log_error() { - echo "$(date +'%F %T') ERROR -- $*" >&2 + echo "$(date +'%F %T') [ERROR] -- $*" >&2 } -- 2.23.0

1 0

[PATCH lkp-tests] feat: support a new field for local disk: rootfs_disk
by Yu Chuan 26 Jan '21

26 Jan '21

[Why] We now need auto install iso to the specific disk of testbox, so in order to avoid conflict with rootfs_partition, we support a new field for local disk: rootfs_disk. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- filters/run_on_local_disk | 8 ++++---- lib/bootstrap.sh | 23 +++++++++++++++++++++++ lkp-exec/qemu | 7 ++++++- 3 files changed, 33 insertions(+), 5 deletions(-) diff --git a/filters/run_on_local_disk b/filters/run_on_local_disk index 389d1ca17fdc..2092bd2b6028 100755 --- a/filters/run_on_local_disk +++ b/filters/run_on_local_disk @@ -1,11 +1,11 @@ #!/bin/bash -# check if testbox has rootfs_partition to run the job +# check if testbox has rootfs_partition or rootfs_disk to run the job if [[ "$run_on_local_disk" == "yes" || "$run_on_local_disk" == "true" ]]; then - if [[ -z "$rootfs_partition" ]]; then - echo "rm $job_file due to no rootfs_partition" - echo "run_on_local_disk needs rootfs_partition" + if [[ -z "$rootfs_partition" && -z "$rootfs_disk" ]]; then + echo "rm $job_file due to no rootfs_partition or rootfs_disk" + echo "run_on_local_disk needs rootfs_partition or rootfs_disk" exit 1 fi fi diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd48b7..6c0055bd19f5 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -490,6 +490,12 @@ wait_load_disk() ls $rootfs_partition >/dev/null 2>&1 && return # eg: LABEL=LKP-ROOTFS blkid | grep -q ${rootfs_partition#*=} && return + + # eg: /dev/disk/by-id/ata-WDC_WD1002FAEX-00Z3A0_WD-WCATRC577623 + ls $rootfs_disk >/dev/null 2>&1 && return + # eg: LABEL=LKP-ROOTFS + blkid | grep -q ${rootfs_disk#*=} && return + sleep 1 done @@ -515,6 +521,23 @@ mount_rootfs() mkdir -p $ROOTFS_DIR/tmp CACHE_DIR=$ROOTFS_DIR/tmp cleanup_pkg_cache $CACHE_DIR + elif [ -n "$rootfs_disk" ]; then + # wait for the machine to load the disk + wait_load_disk || { + # skipping following test if disk can't be load + echo "can't load the disk $rootfs_disk, skip testing..." + set_job_state 'load_disk_fail' + return 1 + } + ROOTFS_DIR=/opt/rootfs + mkdir -p $ROOTFS_DIR + mount $rootfs_disk $ROOTFS_DIR || { + mkfs.btrfs -f $rootfs_disk + mount $rootfs_disk $ROOTFS_DIR + } + mkdir -p $ROOTFS_DIR/tmp + CACHE_DIR=$ROOTFS_DIR/tmp + cleanup_pkg_cache $CACHE_DIR else CACHE_DIR=/tmp/cache mkdir -p $CACHE_DIR diff --git a/lkp-exec/qemu b/lkp-exec/qemu index 9e4b576cf50f..1cbd316f6146 100755 --- a/lkp-exec/qemu +++ b/lkp-exec/qemu @@ -91,14 +91,16 @@ replace_script_partition_val() local nr_hdd_vdisk="$(echo $hdd_partitions | wc -w)" local nr_ssd_vdisk="$(echo $ssd_partitions | wc -w)" local nr_swap_vdisk="$(echo $swap_partitions | wc -w)" + local nr_rootfs_vdisk="$(echo $rootfs_partition | wc -w)" + [ -z "$rootfs_partition" ] && nr_rootfs_vdisk="$(echo $rootfs_disk | wc -w)" [[ $nr_hdd_partitions ]] || nr_hdd_partitions=$nr_hdd_vdisk [[ $nr_ssd_partitions ]] || nr_ssd_partitions=$nr_ssd_vdisk VDISK_NUM=$((nr_hdd_partitions+nr_ssd_partitions+nr_swap_vdisk+nr_rootfs_vdisk)) - [[ "$hdd_partitions$ssd_partitions$swap_partitions$rootfs_partition" =~ '/dev/vda' ]] && return + [[ "$hdd_partitions$ssd_partitions$swap_partitions$rootfs_partition$rootfs_disk" =~ '/dev/vda' ]] && return if ((VDISK_NUM)); then local index=0 @@ -115,6 +117,9 @@ replace_script_partition_val() -e "s%export swap_partitions=.*%export swap_partitions='${vdisk_swap_val}'%" \ -e "s%export rootfs_partition=.*%export rootfs_partition='${vdisk_rootfs_val}'%" \ $job_script + + [ -z "$rootfs_partition" ] && + sed -i -e "s%export rootfs_disk=.*%export rootfs_disk='${vdisk_rootfs_val}'%" $job_script fi } -- 2.23.0

3 5

[PATCH compass-ci] update apply account email address
by Li Yuanchao 26 Jan '21

26 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI �� git repos�� **��** -1. �� compass-ci-reply(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� +1. �� compass-ci-robot(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� 2. ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9C%AC%E5%9C%B0%E5%AE%89%E8%A3%85compass-ci%E5%AE%A2%E6%88%B7%E7%AB%AF.md)�� 3. �� sshd ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%99%BB%E5%BD%95%E6%B5%8B%E8%AF%95%E6%9C%BA%E8%B0%83%E6%B5%8B%E4%BB%BB%E5%8A%A1.md)�� -- 2.23.0

1 0

[PATCH v13 compass-ci] lib/es_jobs.rb: optimize es-jobs output
by Lu Kaiyi 26 Jan '21

26 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archlinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archlinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.summary.any_error=1: 40 kvcount.summary.any_stderr=1: 44 kvcount.summary.any_warning=1: 23 kvcount.summary.success=1: 1 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 ... raw.id.[job_state=failed]: ["crystal.122405","crystal.122406","crystal.122331","crystal.122365","crystal.122404","crystal.122438","crystal.122607","crystal.122370","crystal.122375","crystal.122410","crystal.122900","crystal.122906","crystal.122271","crystal.122451","crystal.122835","crystal.122834","crystal.122232","crystal.122403","crystal.122453","crystal.122886"] ... raw.id.[summary.any_error=1]: ["crystal.122436","crystal.122476","crystal.122803","crystal.122883","crystal.122910","crystal.122438","crystal.122452","crystal.122895","crystal.122446","crystal.122685","crystal.122687","crystal.122833","crystal.122451","crystal.122690","crystal.122719","crystal.122882","crystal.122890","crystal.122453","crystal.122443","crystal.122435","crystal.122432","crystal.122688","crystal.122689","crystal.122886"] ... sum.stats.last_state.test.iperf.exit_code.127: 12 sum.stats.stderr./lkp/lkp/src/lib/env.sh:#:set_perf_path: 12 sum.stats.last_state.is_incomplete_run: 12 sum.stats.kmsg.timestamp:last: 45 sum.stats.last_state.exit_fail: 20 sum.stats.stderr.has_stderr: 44 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0,null,null,null,null,null,null,null,null,524288.0,524288.0,null,null,null,null,null,516608.0,null,1048576.0,null,null,null,null,null,null,null,524288.0,null,null,null,null,2104832.0,1572864.0,null,null,2097152.0,null,null,null,null,null,null,null,null,null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... max.stats.sched_debug.cfs_rq:/.load.stddev: 2104832.0 max.stats.softirqs.CPU1.NET_RX: 6 max.stats.slabinfo.kmalloc-512.active_objs: 1616 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 213 ++++++++++++++++++++++++++++--------------------- 1 file changed, 123 insertions(+), 90 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..c35b182 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -3,137 +3,170 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo summary.success + summary.any_fail summary.any_error summary.any_stderr summary.any_warning].freeze + +require 'json' require "#{LKP_SRC}/lib/stats" require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @es_query = es_query - @es = ESQuery.new(ES_HOST, ES_PORT) + @jobs = query_jobs_from_es(es_query) @refine = my_refine @fields = fields @stats_filter = stats_filter - @stats_filter_result = {} @refine_jobs = [] - @jobs = {} - @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' - } - set_defaults - deal_jobs + set_jobs_summary end - def set_defaults - query_result = @es.multi_field_query(@es_query) - query_result['hits']['hits'].each do |job| - @jobs[job['_id']] = job['_source'] - end - - @stats = { - 'stats.count' => Hash.new(0), - 'stats.sum' => Hash.new(0), - 'stats.avg' => Hash.new(0) - } - @result = {} - @fields.each do |field| - @result[field] = [] - end + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs end - def add_result_fields(job, level) - return unless @refine.include?(level) || @refine.include?(-1) - - @refine_jobs << job['id'] - @fields.each do |field| - value = job[field] - if value - value = job['id'] + '.' + value if field == 'job_state' - @result[field] << value + def set_job_summary(stats, job) + summary_result = '' + stats.each_key do |stat| + # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, + # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, + # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, + if stat.match(/warning/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "stderr.linux-perf": 1, + # "stderr.error:target_not_found:ruby-dev": 1, + # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, + if stat.match(/stderr\./i) + job['summary.any_stderr'] = 1 + summary_result = 'stderr' + end + # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, + # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, + # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + # "last_state.test.iperf.exit_code.127": 1, + # "last_state.test.cci-makepkg.exit_code.1": 1, + if stat.match(/error|\.exit_code\./i) + job['summary.any_error'] = 1 + summary_result = 'error' + end + if stat.match(/\.fail$/i) + job['summary.any_fail'] = 1 + summary_result = 'fail' end - - next unless job['stats'] - - @result[field] << job['stats'][field] if job['stats'][field] end - end - - def deal_jobs - stats_count = Hash.new(0) - stats_jobs = {} + return unless summary_result.empty? - @jobs.each do |job_id, job| - level = deal_stats(job) - add_result_fields(job, level) + job['summary.success'] = 1 + end - stat_key = @stats_level[level] - stat_jobs_key = stat_key + '_jobs' + # set jobs summary fields information in place + def set_jobs_summary + @jobs.each do |job| + stats = job['stats'] + next unless stats - stats_count[stat_key] += 1 - stats_jobs[stat_jobs_key] ||= [] - stats_jobs[stat_jobs_key] << job_id + set_job_summary(stats, job) end - - @stats['stats.count'].merge!(stats_count) - @stats['stats.count'].merge!(stats_jobs) end - def deal_stats(job, level = 0) - return 1 unless job['stats'] + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats - job['stats'].each do |key, value| - match_stats_filter(key, value, job['id']) - calculate_stat(key, value) - level = get_stat_level(key, level) + metrics.concat(stats.keys) end - return level + metrics.uniq! end - def match_stats_filter(key, value, job_id) - @stats_filter.each do |filter| - next unless key.include?(filter) + def initialize_result_hash(metrics) + result = { + 'kvcount' => {}, + 'raw.id' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} + } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end - key = job_id + '.' + key - @stats_filter_result[key] = value + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - break + stats.each do |key, value| + result['raw.stats'][key] << value end end - def calculate_stat(key, value) - if function_stat?(key) - return unless @fields.include?('stats.sum') + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats['stats.sum'][key] += value - else - return unless @fields.include?('stats.avg') + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] + result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] + end + end - @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end end end - def get_stat_level(stat, level) - return level if level >= 3 - return 3 if stat.match(/error|fail/i) - return 2 if stat.match(/warn/i) + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end - return 0 + stats_count(result) + result end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def output_yaml(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + if value.is_a? Hash + output_yaml(prefix_key, value) + else + puts "#{prefix_key}: #{value.to_json}" + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state(@jobs) + @result['kvcount'] = @result['kvcount'].sort.to_h + @result['raw.id'] = @result['raw.id'].sort.to_h + output_yaml('', @result) end end -- 2.23.0

2 2

[PATCH compass-ci] lib/git: use work_dir replace repo for GitCommit class
by Cao Xueliang 26 Jan '21

26 Jan '21

If we use "git -C /srv/git/#{repo}.git", we need mount "/srv/git", but sometimes we have clone the git repo, so we don't need mount "/srv/git". Now, we just need to pass the work_dir to the GitCommit class. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git.rb | 4 ++-- lib/git_bisect.rb | 2 +- lib/mail_bisect_result.rb | 8 +++++++- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/lib/git.rb b/lib/git.rb index db97c47..9751dcf 100644 --- a/lib/git.rb +++ b/lib/git.rb @@ -2,8 +2,8 @@ # wrap common git commands class GitCommit - def initialize(repo, commit) - @git_prefix = "git -C /srv/git/#{repo}.git" + def initialize(work_dir, commit) + @git_prefix = "git -C #{work_dir}" @commit = commit end diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index e577afd..5d13074 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -100,10 +100,10 @@ class GitBisect bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_error_id(@error_id) all_errors = Utils.obt_errors(@work_dir, first_bad_commit) first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) - FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, + 'work_dir' => @work_dir, 'bisect_error' => bisect_error, 'all_errors' => all_errors, 'pkgbuild_repo' => @pkgbuild_repo, diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 209a28d..637c008 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -12,12 +12,13 @@ require_relative 'assign_account_client' class MailBisectResult def initialize(bisect_info) @repo = bisect_info['repo'] + @work_dir = bisect_info['work_dir'] @commit_id = bisect_info['commit'] @all_errors = bisect_info['all_errors'] @bisect_error = bisect_info['bisect_error'] @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] - @git_commit = GitCommit.new(@repo, @commit_id) + @git_commit = GitCommit.new(@work_dir, @commit_id) @to = @git_commit.author_email # now send mail to review @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' @@ -26,6 +27,7 @@ class MailBisectResult def create_send_email send_report_mail(compose_mail) send_account_mail + rm_work_dir end def compose_mail @@ -80,4 +82,8 @@ class MailBisectResult apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end + + def rm_work_dir + FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) + end end -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs: fix error when search nothing
by Wu Zhende 26 Jan '21

26 Jan '21

[Before] es-jobs suite=test Traceback (most recent call last): 3: from /home/wuzhende/compass-ci/sbin/es-jobs:55:in `<main>' 2: from /home/wuzhende/compass-ci/lib/es_jobs.rb:167:in `output' 1: from /home/wuzhende/compass-ci/lib/es_jobs.rb:137:in `query_jobs_state' /home/wuzhende/compass-ci/lib/es_jobs.rb:99:in `initialize_result_hash': undefined method `each' for nil:NilClass (NoMethodError) [After] es-jobs suite=test No query result is found: {"suite"=>"test"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index c35b182..eb3f70d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -13,7 +13,8 @@ require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @jobs = query_jobs_from_es(es_query) + @es_query = es_query + @jobs = query_jobs_from_es @refine = my_refine @fields = fields @stats_filter = stats_filter @@ -21,9 +22,9 @@ class ESJobs set_jobs_summary end - def query_jobs_from_es(items) + def query_jobs_from_es es = ESQuery.new(ES_HOST, ES_PORT) - result = es.multi_field_query items + result = es.multi_field_query @es_query jobs = result['hits']['hits'] jobs.map! { |job| job['_source'] } return jobs @@ -164,6 +165,10 @@ class ESJobs end def output + if @jobs.empty? + puts "No query result is found: #{@es_query}" + return + end @result = query_jobs_state(@jobs) @result['kvcount'] = @result['kvcount'].sort.to_h @result['raw.id'] = @result['raw.id'].sort.to_h -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/apply-account: the repo should be in our test coverage
by Wu Fengguang 26 Jan '21

26 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index b5430e7..3e253d0 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -27,8 +27,9 @@ Email template: - please replace the 3 parts in {{ }} with your information. - the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. -- my oss commit: a git commit URL that has your contribution. It should be in a - well known git repo that's registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos) +- my oss commit: a git commit URL that has your name and email. + The git repo should be in our test coverage and registered in + [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). ## step 2: receive an email -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci January 2021