- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] scheduler: fix block error
by Ren Wen 23 Dec '20

23 Dec '20

[why] Every request to scheduler will initialize a new 'Sched', each 'Sched' will new a 'BlockHelper'. For each node in cluster, it can't release when blocking at scheduler because they use different locks. [how] Set an instance of 'BlockHelper' to class variable. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 3 ++- src/scheduler/request_cluster_state.cr | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f7220f9..25e3708 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -28,11 +28,12 @@ class Sched property redis property block_helper + @@block_helper = BlockHelper.new + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new - @block_helper = BlockHelper.new @rgc = RemoteGitClient.new @env = env @log = env.log.as(JSONLogger) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 1782797..0703792 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -25,7 +25,7 @@ class Sched update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - @block_helper.block_until_finished(cluster_id) { + @@block_helper.block_until_finished(cluster_id) { cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) cluster_state == "ready" || cluster_state == "abort" } -- 2.23.0

1 0

[PATCH lkp-tests] Makefile: delete set-env in Makefile
by Wei Jihui 23 Dec '20

23 Dec '20

set-env is move to install-dependencies.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 0b2d09a2..236bea25 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh bash sbin/install-dependencies.sh .PHONY: doc -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: capturing exceptions
by Wu Zhende 23 Dec '20

23 Dec '20

For API-level errors, use warn. For service-level errors, use error. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 14 ++++++++++++++ src/scheduler.cr | 9 ++++++++- src/scheduler/close_job.cr | 2 ++ src/scheduler/download_file.cr | 2 ++ src/scheduler/find_job_boot.cr | 2 ++ src/scheduler/find_next_job_boot.cr | 2 ++ src/scheduler/request_cluster_state.cr | 2 ++ src/scheduler/update_job_parameter.cr | 2 ++ 8 files changed, 34 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f7220f9..be8cac5 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -45,6 +45,8 @@ class Sched def alive(version) debug_message("Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) end def normalize_mac(mac : String) @@ -59,6 +61,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def del_host_mac @@ -69,6 +73,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def set_host2queues @@ -79,6 +85,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def del_host2queues @@ -89,6 +97,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def update_tbox_wtmp @@ -118,6 +128,8 @@ class Sched # json log hash["testbox"] = testbox @log.info(hash.to_json) + rescue e + @log.warn(e) end def report_ssh_port @@ -130,6 +142,8 @@ class Sched end @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) + rescue e + @log.warn(e) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/scheduler.cr b/src/scheduler.cr index 41702e7..07b38a3 100644 --- a/src/scheduler.cr +++ b/src/scheduler.cr @@ -3,7 +3,14 @@ require "scheduler/scheduler" require "./scheduler/constants.cr" +require "./lib/json_logger" module Scheduler - Kemal.run(SCHED_PORT) + log = JSONLogger.new + + begin + Kemal.run(SCHED_PORT) + rescue e + log.error(e) + end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d8f80d7..9b4da6a 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -24,5 +24,7 @@ class Sched @redis.remove_finished_job(job_id) @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + rescue e + @log.warn(e) end end diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr index e1f0c9a..50e27ba 100644 --- a/src/scheduler/download_file.cr +++ b/src/scheduler/download_file.cr @@ -10,5 +10,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "download"})) send_file @env, file_path + rescue e + @log.warn(e) end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9b3db88..5f71b32 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -21,6 +21,8 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id response + rescue e + @log.warn(e) end # auto submit a job to collect the host information diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 0bdfe5b..091c41a 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -14,5 +14,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id response + rescue e + @log.warn(e) end end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 1782797..06e3329 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -68,6 +68,8 @@ class Sched # show cluster state return @redis.hash_get("sched/cluster_state", cluster_id) + rescue e + @log.warn(e) end # node_state: "finish" | "ready" diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index da528c1..744274a 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -30,5 +30,7 @@ class Sched log = job_content.dup log["job_id"] = log.delete("id").not_nil! @log.info(log.to_json) + rescue e + @log.warn(e) end end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 23 Dec '20

23 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml ��[YAML](http://yaml.org/YAM 4) �� archlinux �� PKGBUILD �� PKGBUILD �� - PKGBUILD �� /srv/git/archlinux/*/*/PKGBUILD�� + PKGBUILD �� `/srv/git/archlinux/*/*/PKGBUILD`�� ### depends -- 2.23.0

1 0

[PATCH v6 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 80 +++++++++++++------------------------------------ 1 file changed, 21 insertions(+), 59 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..b8207ba 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,78 +150,40 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 23 Dec '20

23 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..baf936c 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

1 0

[PATCH compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 23 Dec '20

23 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..336602e 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

2 2

Re: [PATCH lkp-tests] Makefile: delete set-env in makefile
by Liu Shaofei 22 Dec '20

22 Dec '20

Reply-To: In-Reply-To: <20201222125413.2442750-1-weijihuiall(a)163.com> The Titel: makefile => Makefile? Thanks, Liushaofei On Tue, Dec 22, 2020 at 08:54:13PM +0800, Wei Jihui wrote: >set-env is move to install-dependencies.sh > >Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >--- > Makefile | 1 - > 1 file changed, 1 deletion(-) > >diff --git a/Makefile b/Makefile >index 0b2d09a2..236bea25 100644 >--- a/Makefile >+++ b/Makefile >@@ -14,7 +14,6 @@ subsystem: > install: > mkdir -p $(TARGET_DIR_BIN) > ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp >- bash sbin/set-env.sh > bash sbin/install-dependencies.sh > > .PHONY: doc >-- >2.23.0 >

1 0

[PATCH compass-ci] result-webdav: if judgment error and some file symlinks are not created
by Cui Yili 22 Dec '20

22 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index f077fca..1feddd4 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -46,7 +46,7 @@ http { link_name = "latest" else tmp_name = string.match(upload_file, "(.*)_%d+%.cgz") - if not tmp_name then + if tmp_name then link_name = tmp_name else return -- 2.23.0

2 1

[PATCH lkp-tests] Makefile: delete set-env in makefile
by Wei Jihui 22 Dec '20

22 Dec '20

set-env is move to install-dependencies.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 0b2d09a2..236bea25 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh bash sbin/install-dependencies.sh .PHONY: doc -- 2.23.0

1 0