- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] lib/job.rb: fix job symbol key check
by Wei Jihui 23 Dec '20

23 Dec '20

load job from yaml maybe has symbol key symbol key does not has start_with function. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7553312e..5ae67e6b 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,7 +254,6 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end @@ -289,7 +288,12 @@ class Job def multi_args(hash) jobs_array = [] hash.each { |key, value| - next unless key =~ /^\w.*\|.*\w$/ + next unless key =~ /^\w.*\|.*\w$/ || key.is_a?(String) + + if key.start_with?('#! ') + hash.delete(key) + next + end key_array = key.split('|').map(&:strip) [value].flatten.each do |v| -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..7274e2e 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 0

[PATCH compass-ci] docker-rootf: add different install methods
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..6a121d3 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) ]; then + apt update + apt-get install -y --no-install-recommends $(</tmp/packages-to-install) + elif [ $(command -v pacman) ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

2 2

[PATCH lkp-tests] lkp-tests: delete useless file
by Wei Jihui 23 Dec '20

23 Dec '20

install.sh is useless now install method is: make install && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 install.sh diff --git a/install.sh b/install.sh deleted file mode 100644 index cea1654f..00000000 --- a/install.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash - -. $LKP_SRC/lib/env.sh - -make - -source $(shell_profile) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix block error
by Ren Wen 23 Dec '20

23 Dec '20

[why] Every request to scheduler will initialize a new 'Sched', each 'Sched' will new a 'BlockHelper'. For each node in cluster, it can't release when blocking at scheduler because they use different locks. [how] Set an instance of 'BlockHelper' to class variable. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 3 ++- src/scheduler/request_cluster_state.cr | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f7220f9..25e3708 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -28,11 +28,12 @@ class Sched property redis property block_helper + @@block_helper = BlockHelper.new + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new - @block_helper = BlockHelper.new @rgc = RemoteGitClient.new @env = env @log = env.log.as(JSONLogger) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 1782797..0703792 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -25,7 +25,7 @@ class Sched update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - @block_helper.block_until_finished(cluster_id) { + @@block_helper.block_until_finished(cluster_id) { cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) cluster_state == "ready" || cluster_state == "abort" } -- 2.23.0

1 0

[PATCH lkp-tests] Makefile: delete set-env in Makefile
by Wei Jihui 23 Dec '20

23 Dec '20

set-env is move to install-dependencies.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 0b2d09a2..236bea25 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh bash sbin/install-dependencies.sh .PHONY: doc -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: capturing exceptions
by Wu Zhende 23 Dec '20

23 Dec '20

For API-level errors, use warn. For service-level errors, use error. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 14 ++++++++++++++ src/scheduler.cr | 9 ++++++++- src/scheduler/close_job.cr | 2 ++ src/scheduler/download_file.cr | 2 ++ src/scheduler/find_job_boot.cr | 2 ++ src/scheduler/find_next_job_boot.cr | 2 ++ src/scheduler/request_cluster_state.cr | 2 ++ src/scheduler/update_job_parameter.cr | 2 ++ 8 files changed, 34 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f7220f9..be8cac5 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -45,6 +45,8 @@ class Sched def alive(version) debug_message("Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) end def normalize_mac(mac : String) @@ -59,6 +61,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def del_host_mac @@ -69,6 +73,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def set_host2queues @@ -79,6 +85,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def del_host2queues @@ -89,6 +97,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def update_tbox_wtmp @@ -118,6 +128,8 @@ class Sched # json log hash["testbox"] = testbox @log.info(hash.to_json) + rescue e + @log.warn(e) end def report_ssh_port @@ -130,6 +142,8 @@ class Sched end @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) + rescue e + @log.warn(e) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/scheduler.cr b/src/scheduler.cr index 41702e7..07b38a3 100644 --- a/src/scheduler.cr +++ b/src/scheduler.cr @@ -3,7 +3,14 @@ require "scheduler/scheduler" require "./scheduler/constants.cr" +require "./lib/json_logger" module Scheduler - Kemal.run(SCHED_PORT) + log = JSONLogger.new + + begin + Kemal.run(SCHED_PORT) + rescue e + log.error(e) + end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d8f80d7..9b4da6a 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -24,5 +24,7 @@ class Sched @redis.remove_finished_job(job_id) @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + rescue e + @log.warn(e) end end diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr index e1f0c9a..50e27ba 100644 --- a/src/scheduler/download_file.cr +++ b/src/scheduler/download_file.cr @@ -10,5 +10,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "download"})) send_file @env, file_path + rescue e + @log.warn(e) end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9b3db88..5f71b32 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -21,6 +21,8 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id response + rescue e + @log.warn(e) end # auto submit a job to collect the host information diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 0bdfe5b..091c41a 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -14,5 +14,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id response + rescue e + @log.warn(e) end end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 1782797..06e3329 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -68,6 +68,8 @@ class Sched # show cluster state return @redis.hash_get("sched/cluster_state", cluster_id) + rescue e + @log.warn(e) end # node_state: "finish" | "ready" diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index da528c1..744274a 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -30,5 +30,7 @@ class Sched log = job_content.dup log["job_id"] = log.delete("id").not_nil! @log.info(log.to_json) + rescue e + @log.warn(e) end end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 23 Dec '20

23 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml ��[YAML](http://yaml.org/YAM 4) �� archlinux �� PKGBUILD �� PKGBUILD �� - PKGBUILD �� /srv/git/archlinux/*/*/PKGBUILD�� + PKGBUILD �� `/srv/git/archlinux/*/*/PKGBUILD`�� ### depends -- 2.23.0

1 0

[PATCH v6 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 80 +++++++++++++------------------------------------ 1 file changed, 21 insertions(+), 59 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..b8207ba 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,78 +150,40 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 23 Dec '20

23 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..baf936c 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

1 0