- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/4] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 28 Sep '20

28 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try aggin later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..6bd7f21 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -31,3 +31,6 @@ export CCI_SRC=/c/compass-ci export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

2 1

[PATCH compass-ci] Change crystal-ci to compass-ci
by Zhang Yu 28 Sep '20

28 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index d366580..d9f6158 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1,4 +1,4 @@ -# Crystal-CI Introduction +# Compass-CI Introduction ## 简单说明 @@ -659,7 +659,7 @@ Compass-CI 集开发调测、测试服务、测试结果分析、辅助定位为 b) 将测试用例所需debian下的包名填入创建的文件中 echo "apache2" >> lkp-tests/distro/depends/test c) 生成lkp-aarch64.cgz - ./crystal-ci/container/lkp-initrd/run + ./compass-ci/container/lkp-initrd/run d) 提交测试用例 submit test.yaml e) 查看结果 -- 2.23.0

2 1

[PATCH compass-ci 2/4] sparrow: use useradd to add a lkp user instead of adduser
by Hu Xuejiao 28 Sep '20

28 Sep '20

Executing "adduser -u 1090 lkp" appear: New password: After usering "adduser ** --disabled-password **" appear: Full Name []: So use "adduser" to need too much user-information. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 1bb6861..05becdf 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -4,7 +4,7 @@ gem install git activesupport rest-client cucumber json faye-websocket elasticsearch -grep -q '^lkp:' /etc/passwd || adduser -u 1090 lkp +grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^team:' /etc/group || groupadd team grep -q '^committer:' /etc/group || groupadd -g 1999 committer -- 2.23.0

1 0

[PATCH v2 compass-ci] src/extract-stats prefix exception handing
by Lu Weitao 28 Sep '20

28 Sep '20

background: extract-stats process is at risk of be interrupted, due to other dependent services(task-queue, es_client) may not working properly solution: 1. use begin ... rescue to handle possible exceptions 2. skip and remove the task from task-queue if the job data does not have post-processing conditions 3. refactor for stats_worker.cr Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 60 ++++++++++++++++++------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..edebf27 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -18,40 +18,50 @@ class StatsWorker def consume_sched_queue(queue_path : String) loop do begin - response = @tq.consume_task(queue_path) + job_id = get_job_id(queue_path).to_s + if job_id # consume the job by post-processing + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + end rescue e STDERR.puts e.message - # incase of many error message when task-queue is busy + # incase of many error message when task-queue, ES does not work sleep(10) - next end + end + end - if response[0] == 200 - job_id = JSON.parse(response[1].to_json)["id"] - job = @es.get_job_content(job_id.to_s) - result_root = job["result_root"] - begin - # extract stats.json - system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" - # storage stats to job in es - store_stats_es(result_root.to_s, job_id.to_s) if result_root - # send mail to submitter for job results - system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" - rescue e - STDERR.puts e.message - next - end - - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - else - sleep(60) - end + # get job_id from task-queue + def get_job_id(queue_path : String) + response = @tq.consume_task(queue_path) + if response[0] == 200 + JSON.parse(response[1].to_json)["id"] + else + # will sleep 60s if no task in task-queue + sleep(60) + nil end end - def store_stats_es(result_root : String, job_id : String) + def result_post_processing(job_id : String, result_root : String, queue_path : String) + return nil unless result_root + + # extract stats.json + system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" + # storage stats to job in es + store_stats_es(result_root, job_id, queue_path) + # send mail to submitter for job results + system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" + end + + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - raise "#{stats_path} file not exists." unless File.exists?(stats_path) + unless File.exists?(stats_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + raise "#{stats_path} file not exists." + end stats = File.open(stats_path) do |file| JSON.parse(file) -- 2.23.0

1 0

[PATCH compass-ci] feat: scheduler: add optional fields into job
by Ren Wen 28 Sep '20

28 Sep '20

Only `multi-qemu` job needs `FLUENTD_SERVER_*`, so move them to the place where calls this function. When calls this function, an array can be passed, like: ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}",] these elements will add into job as job fields. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/jobfile_operate.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index df5af1b..0533346 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -200,14 +200,13 @@ module Jobfile::Operate return target_path end - def self.auto_submit_job(job_file, override_parameter) + def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) job_fields = ["SCHED_HOST=#{ENV["SCHED_HOST"]}", "SCHED_PORT=#{ENV["SCHED_PORT"]}", - "FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", - "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}", "GIT_SERVER=#{ENV["GIT_SERVER"]}", "lab=#{ENV["lab"]}"] + job_fields.concat(other_parameters) if other_parameters cmd = "#{ENV["LKP_SRC"]}/sbin/submit " cmd += job_fields.join(" ") cmd += " -s '#{override_parameter}' #{job_file}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow: container: fix git-mirror start error
by Liu Yinsi 28 Sep '20

28 Sep '20

1. git clone upstream-repo and export REPO_SRC to fix error: No such file or directory @ rb_sysopen - /c/upstream-repos 2. chown lkp:committer /srv/git to fix error: Permission denied @ dir_s_mkdir - /srv/git/b Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/3-code/dev-env | 1 + sparrow/3-code/git | 4 ++++ 3 files changed, 6 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 35456a8..f5c1646 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -6,6 +6,7 @@ chown -R lkp:lkp /srv/es chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result chmod 775 /srv/es/logging-es diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..ae267bd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -27,6 +27,7 @@ EOF cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci +export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF diff --git a/sparrow/3-code/git b/sparrow/3-code/git index b1d2c03..5588696 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -16,3 +16,7 @@ cd /c || exit [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git } + +[ -d "upstream-repos/.git" ] || { + git clone https://gitee.com/wu_fengguang/upstream-repos.git +} -- 2.23.0

1 0

[PATCH compass-ci] Makefile: add ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 Makefile diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..c7505e8 --- /dev/null +++ b/Makefile @@ -0,0 +1,2 @@ +ctags: + ctags -R --links=no --langmap=ruby:.rb.cr -- 2.23.0

1 1

[PATCH compass-ci] fix: system rebooting when executed 2-network/br0
by Xu Xijian 28 Sep '20

28 Sep '20

When 2-network/br0 executed, program will hang up fo a while and then go rebooting. The cause is iptables need time to take effect, echo 1 to /proc/sys/net/ipv4/ip_forward to make it taking effect immediately. Logs(openeuler 20.03): [ 161.565451] + 2-network/iptables [ 161.565453] [ 161.568088] ++ ip route get 1.2.3.4 [ 161.568090] ... [ 168.107379] LKP: rebooting Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- sparrow/2-network/iptables | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..0bf1235 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -11,6 +11,8 @@ BR0_SUBNET=172.18.0.0/16 iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE +# To make upon rules taking effect immediately +echo 1 > /proc/sys/net/ipv4/ip_forward systemctl status firewalld | grep -q "running" && { -- 2.23.0

2 1

[PATCH compass-ci] code-spec: refactor friendly principles
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index e783a0c..f805aec 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -7,6 +7,23 @@ code注意事项 - 一个函数, 不要超过10行代码 - 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 +重构友好编码 +============ + +基本原则: +- 预防自己的代码造成他人的几大困难 + - 难以理解不敢下手 + - 不知道怎么找到*所有*关联逻辑 + - 不知道自己不知道 (unknown unknown) +- 设想将来做重构, 或者需要理解任何一个高层次的功能/流程, + 一个新人应当可以通过 git grep 一两个关键字或者pattern, + 就能找到所有相关底层代码与文档, 而不必担心有所遗漏. + +基本方法: +- 隐式场景显式化: 加注释明确代码的各类隐含目的, 意义, 场景, 数据示例等等 +- 隐式依赖显式化: 定义相同的ID, 把所有的关联文档与各处代码串起来 +- 分散依赖集中化: 定义类/函数/assert, 封装/强制各类设计中的假设, 约定与规则 + PATCH注意事项 ============= -- 2.23.0

1 0

[PATCH lkp-tests] Makefile: support ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Makefile b/Makefile index c40d914b..37387809 100644 --- a/Makefile +++ b/Makefile @@ -23,3 +23,6 @@ doc: tests/%.md: tests/%.yaml lkp gen-doc $< + +ctags: + ctags -R --links=no -- 2.23.0

1 0