December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests 1/2] lib/job.rb: load config from lab yaml
by Wei Jihui 15 Dec '20

15 Dec '20

[why] my_token is per-lab account, so it will load my_token from ~/.config/compass-ci/include/lab/$lab.yaml and $lab is from ~/.config/compass-ci/defaults/*.yaml or /etc/compass-ci/defaults/*.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index b41d55a7..9eac4001 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -414,6 +414,17 @@ class Job true end + def load_self_config + self_config_path = "#{ENV['HOME']}/.config/compass-ci" + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + "#{self_config_path}/defaults/*.yaml"]).each do |file| + load_one_defaults(file, @job) + end + + lab_yaml = File.join(self_config_path, 'include/lab', "#{@defaults['lab']}.yaml") + load_one_defaults(lab_yaml, @job) + end + def load_defaults(first_time = true) if @job.include? :no_defaults merge_defaults first_time @@ -426,11 +437,7 @@ class Job @file_loaded ||= {} end - Dir.glob(["/etc/compass-ci/defaults/*.yaml", - "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| - load_one_defaults(file, @job) - end - + load_self_config i = include_files job = deepcopy(@job) job['___'] = nil -- 2.23.0

1 0

[PATCH v4 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 15 Dec '20

15 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 87 ++++++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 41 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..3110685 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -152,29 +152,20 @@ job yaml ��[YAML](http://yaml.org/YAM 1. �� os �� rootfs�� openEuler �� - �� docker �� rootfs - + 1) �� openEuler �� docker �� - + ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` + ``` + 2) {compass-ci}/container/docker-rootfs �� '.tar.xz' �� docker �� rootfs + ```bash + ./{compass-ci}/container/docker-rootfs/run openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs-20.03 + ``` - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� + + 1) �� openEuler �� qcow2 �� ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… ``` @@ -183,45 +174,59 @@ job yaml ��[YAML](http://yaml.org/YAM cd {compass-ci}/container/qcow2rootfs ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) + + 2. ��rootfs + 1. �� chroot �� rootfs (�� root ��) ```bash chroot openEuler-rootfs ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 2. �� + a. �� /sbin/init + > ��systemd�� + ```bash + ls -l /sbin/init + /sbin/init -> ../lib/systemd/systemd + ``` + b. �� /etc/localtime + > �� + ```bash + ls -l /etc/localtime + /etc/localtime -> ../usr/share/zoneinfo/Asia/Shanghai + ``` + c. �� docker �� osimage �� + > docker �� /.dockerenv �� + + 3. ��rootfs + ```bash + cd $rootfs + find . | coip -o -Hnewc |gzip -9 > $os_name.cgz + ``` + + 4. ��/�� + �� {compass-ci} �� rootfs ��/�� + ```bash + find /lib/modules/* | cpio -o -Hnewc | gzip -9 > /modules-$os_name.cgz + find /usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > /headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow: fix that 0-package/os/openEuler can only be executed once
by Yu Chuan 15 Dec '20

15 Dec '20

[Error Msg] ++ rpm -i lib64lockfile1-1.16-1-omv4002.aarch64.rpm lockfile-progs-0.1.18-5-omv4002.aarch64.rpm warning: lib64lockfile1-1.16-1-omv4002.aarch64.rpm: Header V4 RSA/SHA256 Signature, key ID bf81de15: NOKEY package lib64lockfile1-1.16-1.aarch64 is already installed file /usr/lib64/liblockfile.so.1.0 from install of lib64lockfile1-1.16-1.aarch64 conflicts with file from package lib64lockfile1-1.16-1.aarch64 package lockfile-progs-0.1.18-5.aarch64 is already installed file /usr/bin/lockfile-check from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-create from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-remove from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-touch from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-lock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-touchlock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-unlock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/os/openEuler | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 43e96ab8ab2e..5b32c0a23784 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,17 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +install_rpm() +{ + local pkg=$1 + local pkg_url=$2 + + rpm -qa | grep -q "$pkg" && return 0 + + wget $pkg_url + rpm -i $(basename $pkg_url) && rm -rf $(basename $pkg_url) +} + pkgs=( vim git @@ -24,8 +35,10 @@ pkgs=( yum install -y --skip-broken "${pkgs[@]}" -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… +install_rpm \ + "lib64lockfile1-1.16-1.$(arch)"\ + "http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas…" -rpm -i lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm -rm -rf *.rpm +install_rpm \ + "lockfile-progs-0.1.18-5.$(arch)"\ + "http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas…" -- 2.23.0

2 3

[PATCH compass-ci 1/2] sparrow: refactor the code for docker build all image concurrently
by Yu Chuan 15 Dec '20

15 Dec '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/4-docker/buildall | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index eb245008ad45..43a3a06e4222 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -8,18 +8,13 @@ export action=$1 build_depends() { local container=$1 - local should_wait=$2 for dep in $(cat $container/*-depends 2> /dev/null) do - build_depends $CONTAINER_PATH/$dep block_wait + build_depends $CONTAINER_PATH/$dep done - if [ -n "$should_wait" ]; then - do_one $container - else - do_one $container & - fi + do_one $container } do_one() @@ -44,7 +39,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir + build_depends $dir & done wait -- 2.23.0

3 3

[PATCH v3 compass-ci 1/2] mail-robot: use key my_token instead of my_uuid
by Luan Shengde 15 Dec '20

15 Dec '20

[why] key my_uuid is renamed to my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 31 +++++++++++++---------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 0fff3b5..aea1aa5 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -30,14 +30,14 @@ SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 # my_info: # - my_email # - my_name -# - my_uuid +# - my_token # my_ssh_pubkey # store_account_info # call ESClient to store my_info # my_info: # - my_email # - my_name -# - my_uuid +# - my_token # - my_commit_url # - my_login_name # - my_ssh_pubkey @@ -87,21 +87,28 @@ class ApplyAccount accountx_es.query_by_id(@my_info['my_email']) end + def build_apply_info(apply_info, my_account_es) + my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? + apply_info.update my_account_es + apply_info.update @my_info + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + @my_info.update apply_info + apply_info['is_update_account'] = true + apply_info + end + def apply_my_account my_account_es = read_my_account_es apply_info = {} if my_account_es - my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') - apply_info.update my_account_es - apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq - @my_info.update apply_info - apply_info['is_update_account'] = true + build_apply_info(apply_info, my_account_es) else - my_uuid = %x(uuidgen).chomp - @my_info['my_uuid'] = my_uuid + my_token = %x(uuidgen).chomp + @my_info['my_token'] = my_token apply_info.update @my_info end + apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -112,10 +119,6 @@ class ApplyAccount @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es end - def check_account_es - ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) - end - def store_account_info es = ESClient.new(index: 'accounts') es.put_source_by_id(@my_info['my_email'], @my_info) -- 2.23.0

2 2

[PATCH compass-ci] lib/job.cr: optimize check account error message
by Wu Zhende 15 Dec '20

15 Dec '20

Like: submit iperf.yaml failed, got job_id=0, error: Failed to verify the account. Please refer to https://gitee.com/wu_fengguang/compass-ci/doc/manual/apply-account.md Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0083a7d..7eb8498 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -326,7 +326,8 @@ class Job end private def check_account_info - error_msg = "Failed to verify the account. Please check your configuration" + error_msg = "Failed to verify the account. " + error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/doc/manual/apply-account.md" account_info = @es.get_account(self["my_email"]) raise account_info unless account_info.is_a?(JSON::Any) -- 2.23.0

2 1

[PATCH lkp-tests] iso2rootfs: fix `has_cmd: command not found` error
by Yu Chuan 15 Dec '20

15 Dec '20

[Error Msg] ++ has_cmd yum /lkp/lkp/src/tests/iso2rootfs: line 213: has_cmd: command not found Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 8625b6484d89..9f2a77de17c6 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -6,6 +6,8 @@ set -e umask 0002 +. $LKP_SRC/lib/env.sh + # allow sut os array # - if job's os not in this array, job won't execute ALLOW_OS=( -- 2.23.0

3 2

[PATCH v2 lkp-tests 1/2] lib/job.rb: load config from lab yaml
by Wei Jihui 15 Dec '20

15 Dec '20

[why] my_uuid is per-lab account, so it will load my_uuid from ~/.config/compass-ci/include/lab/$lab.yaml and $lab is from ~/.config/compass-ci/defaults/*.yaml or /etc/compass-ci/defaults/*.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index b41d55a7..9eac4001 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -414,6 +414,17 @@ class Job true end + def load_self_config + self_config_path = "#{ENV['HOME']}/.config/compass-ci" + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + "#{self_config_path}/defaults/*.yaml"]).each do |file| + load_one_defaults(file, @job) + end + + lab_yaml = File.join(self_config_path, 'include/lab', "#{@defaults['lab']}.yaml") + load_one_defaults(lab_yaml, @job) + end + def load_defaults(first_time = true) if @job.include? :no_defaults merge_defaults first_time @@ -426,11 +437,7 @@ class Job @file_loaded ||= {} end - Dir.glob(["/etc/compass-ci/defaults/*.yaml", - "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| - load_one_defaults(file, @job) - end - + load_self_config i = include_files job = deepcopy(@job) job['___'] = nil -- 2.23.0

2 1

[PATCH compass-ci 6/6] fetch-mail: start file
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/start | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/fetch-mail/start diff --git a/container/fetch-mail/start b/container/fetch-mail/start new file mode 100755 index 0000000..372847b --- /dev/null +++ b/container/fetch-mail/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'fetch-mail' + +names = Set.new %w[ + FETCH_MAIL_ADDRESS + FETCH_MAIL_AUTH_CODE + FETCH_MAIL_DIR + FETCH_MAIL_BOX +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + -u mailer + --name=fetch-mail + -d +] + env + %W[ + -v #{defaults['FETCH_MAIL_DIR']}:#{defaults['FETCH_MAIL_DIR']}:rw + -w /home/mailer/ + fetch-mail +] + +cmd += ['sh', '-c', './run.sh'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci 5/6] fetch-mail: build file
by Luan Shengde 15 Dec '20

15 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/fetch-mail/build diff --git a/container/fetch-mail/build b/container/fetch-mail/build new file mode 100755 index 0000000..e72bed8 --- /dev/null +++ b/container/fetch-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t fetch-mail:latest . -- 2.23.0

1 0