September 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/4] sparrow: use useradd to add a lkp user instead of adduser
by Hu Xuejiao 28 Sep '20

28 Sep '20

Executing "adduser -u 1090 lkp" appear: New password: After usering "adduser ** --disabled-password **" appear: Full Name []: So use "adduser" to need too much user-information. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 1bb6861..05becdf 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -4,7 +4,7 @@ gem install git activesupport rest-client cucumber json faye-websocket elasticsearch -grep -q '^lkp:' /etc/passwd || adduser -u 1090 lkp +grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^team:' /etc/group || groupadd team grep -q '^committer:' /etc/group || groupadd -g 1999 committer -- 2.23.0

1 0

[PATCH v2 compass-ci] src/extract-stats prefix exception handing
by Lu Weitao 28 Sep '20

28 Sep '20

background: extract-stats process is at risk of be interrupted, due to other dependent services(task-queue, es_client) may not working properly solution: 1. use begin ... rescue to handle possible exceptions 2. skip and remove the task from task-queue if the job data does not have post-processing conditions 3. refactor for stats_worker.cr Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 60 ++++++++++++++++++------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..edebf27 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -18,40 +18,50 @@ class StatsWorker def consume_sched_queue(queue_path : String) loop do begin - response = @tq.consume_task(queue_path) + job_id = get_job_id(queue_path).to_s + if job_id # consume the job by post-processing + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + end rescue e STDERR.puts e.message - # incase of many error message when task-queue is busy + # incase of many error message when task-queue, ES does not work sleep(10) - next end + end + end - if response[0] == 200 - job_id = JSON.parse(response[1].to_json)["id"] - job = @es.get_job_content(job_id.to_s) - result_root = job["result_root"] - begin - # extract stats.json - system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" - # storage stats to job in es - store_stats_es(result_root.to_s, job_id.to_s) if result_root - # send mail to submitter for job results - system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" - rescue e - STDERR.puts e.message - next - end - - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - else - sleep(60) - end + # get job_id from task-queue + def get_job_id(queue_path : String) + response = @tq.consume_task(queue_path) + if response[0] == 200 + JSON.parse(response[1].to_json)["id"] + else + # will sleep 60s if no task in task-queue + sleep(60) + nil end end - def store_stats_es(result_root : String, job_id : String) + def result_post_processing(job_id : String, result_root : String, queue_path : String) + return nil unless result_root + + # extract stats.json + system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" + # storage stats to job in es + store_stats_es(result_root, job_id, queue_path) + # send mail to submitter for job results + system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" + end + + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - raise "#{stats_path} file not exists." unless File.exists?(stats_path) + unless File.exists?(stats_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + raise "#{stats_path} file not exists." + end stats = File.open(stats_path) do |file| JSON.parse(file) -- 2.23.0

1 0

[PATCH compass-ci] feat: scheduler: add optional fields into job
by Ren Wen 28 Sep '20

28 Sep '20

Only `multi-qemu` job needs `FLUENTD_SERVER_*`, so move them to the place where calls this function. When calls this function, an array can be passed, like: ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}",] these elements will add into job as job fields. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/jobfile_operate.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index df5af1b..0533346 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -200,14 +200,13 @@ module Jobfile::Operate return target_path end - def self.auto_submit_job(job_file, override_parameter) + def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) job_fields = ["SCHED_HOST=#{ENV["SCHED_HOST"]}", "SCHED_PORT=#{ENV["SCHED_PORT"]}", - "FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", - "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}", "GIT_SERVER=#{ENV["GIT_SERVER"]}", "lab=#{ENV["lab"]}"] + job_fields.concat(other_parameters) if other_parameters cmd = "#{ENV["LKP_SRC"]}/sbin/submit " cmd += job_fields.join(" ") cmd += " -s '#{override_parameter}' #{job_file}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow: container: fix git-mirror start error
by Liu Yinsi 28 Sep '20

28 Sep '20

1. git clone upstream-repo and export REPO_SRC to fix error: No such file or directory @ rb_sysopen - /c/upstream-repos 2. chown lkp:committer /srv/git to fix error: Permission denied @ dir_s_mkdir - /srv/git/b Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/3-code/dev-env | 1 + sparrow/3-code/git | 4 ++++ 3 files changed, 6 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 35456a8..f5c1646 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -6,6 +6,7 @@ chown -R lkp:lkp /srv/es chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result chmod 775 /srv/es/logging-es diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..ae267bd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -27,6 +27,7 @@ EOF cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci +export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF diff --git a/sparrow/3-code/git b/sparrow/3-code/git index b1d2c03..5588696 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -16,3 +16,7 @@ cd /c || exit [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git } + +[ -d "upstream-repos/.git" ] || { + git clone https://gitee.com/wu_fengguang/upstream-repos.git +} -- 2.23.0

1 0

[PATCH compass-ci] Makefile: add ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 Makefile diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..c7505e8 --- /dev/null +++ b/Makefile @@ -0,0 +1,2 @@ +ctags: + ctags -R --links=no --langmap=ruby:.rb.cr -- 2.23.0

1 1

[PATCH compass-ci] fix: system rebooting when executed 2-network/br0
by Xu Xijian 28 Sep '20

28 Sep '20

When 2-network/br0 executed, program will hang up fo a while and then go rebooting. The cause is iptables need time to take effect, echo 1 to /proc/sys/net/ipv4/ip_forward to make it taking effect immediately. Logs(openeuler 20.03): [ 161.565451] + 2-network/iptables [ 161.565453] [ 161.568088] ++ ip route get 1.2.3.4 [ 161.568090] ... [ 168.107379] LKP: rebooting Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- sparrow/2-network/iptables | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..0bf1235 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -11,6 +11,8 @@ BR0_SUBNET=172.18.0.0/16 iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE +# To make upon rules taking effect immediately +echo 1 > /proc/sys/net/ipv4/ip_forward systemctl status firewalld | grep -q "running" && { -- 2.23.0

2 1

[PATCH compass-ci] code-spec: refactor friendly principles
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index e783a0c..f805aec 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -7,6 +7,23 @@ code注意事项 - 一个函数, 不要超过10行代码 - 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 +重构友好编码 +============ + +基本原则: +- 预防自己的代码造成他人的几大困难 + - 难以理解不敢下手 + - 不知道怎么找到*所有*关联逻辑 + - 不知道自己不知道 (unknown unknown) +- 设想将来做重构, 或者需要理解任何一个高层次的功能/流程, + 一个新人应当可以通过 git grep 一两个关键字或者pattern, + 就能找到所有相关底层代码与文档, 而不必担心有所遗漏. + +基本方法: +- 隐式场景显式化: 加注释明确代码的各类隐含目的, 意义, 场景, 数据示例等等 +- 隐式依赖显式化: 定义相同的ID, 把所有的关联文档与各处代码串起来 +- 分散依赖集中化: 定义类/函数/assert, 封装/强制各类设计中的假设, 约定与规则 + PATCH注意事项 ============= -- 2.23.0

1 0

[PATCH lkp-tests] Makefile: support ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Makefile b/Makefile index c40d914b..37387809 100644 --- a/Makefile +++ b/Makefile @@ -23,3 +23,6 @@ doc: tests/%.md: tests/%.yaml lkp gen-doc $< + +ctags: + ctags -R --links=no -- 2.23.0

1 0

[PATCH compass-ci] doc: add code-spec.md
by Wu Fengguang 27 Sep '20

27 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 42 ++++++++++++++++++++++++++++++++++++++ doc/develop-environment.md | 1 + 2 files changed, 43 insertions(+) create mode 100644 doc/code-spec.md diff --git a/doc/code-spec.md b/doc/code-spec.md new file mode 100644 index 0000000..e783a0c --- /dev/null +++ b/doc/code-spec.md @@ -0,0 +1,42 @@ +code注意事项 +============ + +- 代码应当放进函数里不要在脚本顶层直接写逻辑 +- 代码应当放进lib/里脚本本身最小化, 仅解析命令行参数, 调用lib/入口函数 +- 如果出现了全局变量一般意味着需要写一个 class/module 来放这个状态量了 +- 一个函数, 不要超过10行代码 +- 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 + +PATCH注意事项 +============= + +- PATCH title 应当简要说明做了什么 (what) +- PATCH changelog / code comment 应当解释读者无法从代码简单推断的 + - 背景 + - 目的 + - 效果 + - 设计选项和权衡 + - 实际例子 + - 改进数字 + - bug场景, 出错消息原文 +- PATCH 应当加上作者签名 (Signed-off-by: Name <email>) +- PATCH 最好不超过100行, 一个PATCH只做一件事, 避免混杂多个逻辑变更 + +PATCH参考材料 +============= + +The patch subject and changelog should be well written according to suggestions +here: + + https://github.com/thoughtbot/dotfiles/blob/master/gitmessage + https://www.cnblogs.com/cpselvis/p/6423874.html + + http://www.ozlabs.org/~akpm/stuff/tpp.txt + 2: Subject: + 3: Attribution + 4: Changelog + + https://www.kernel.org/doc/html/latest/translations/zh_CN/process/submittin… + 2) 描述你的改动 + 3) 拆分你的改动 + 8）回复评审意见 diff --git a/doc/develop-environment.md b/doc/develop-environment.md index 3af44ac..8760878 100644 --- a/doc/develop-environment.md +++ b/doc/develop-environment.md @@ -50,6 +50,7 @@ Then read through documents lkp-tests/doc/INSTALL.md compass-ci/doc/INSTALL.md + compass-ci/doc/code-spec.md compass-ci/doc/learning-resources.md # crystal compiler -- 2.23.0

1 0

[PATCH todo] credit: fix typos
by Wu Fengguang 26 Sep '20

26 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- people/LiangTao/credit/2020-09-11.md | 2 +- people/LiuShaofei/credit/2020-08-14.md | 2 +- people/LiuYinsi/credit/2020-09-25.md | 1 - people/LuWeitao/credit/2020-09-18.md | 2 +- people/LuWeitao/credit/2020-09-25.md | 2 +- people/LuanShengde/credit/2020-09-18.md | 1 - people/WangChenglong/credit/2020-09-25.md | 1 - people/XuXijian/credit/2020-07-31.md | 2 +- people/XuXijian/credit/2020-08-07.md | 2 +- people/XuXijian/credit/2020-08-14.md | 2 +- 10 files changed, 7 insertions(+), 10 deletions(-) diff --git a/people/LiangTao/credit/2020-09-11.md b/people/LiangTao/credit/2020-09-11.md index 87afa33..464173b 100644 --- a/people/LiangTao/credit/2020-09-11.md +++ b/people/LiangTao/credit/2020-09-11.md @@ -1,4 +1,4 @@ Shi Zhichao Xiao Shenwei Wang Chenglong -Wei Jihiu +Wei Jihui diff --git a/people/LiuShaofei/credit/2020-08-14.md b/people/LiuShaofei/credit/2020-08-14.md index 66d1d24..a5a3f35 100644 --- a/people/LiuShaofei/credit/2020-08-14.md +++ b/people/LiuShaofei/credit/2020-08-14.md @@ -4,4 +4,4 @@ Wang Yong Bai Jing Yu Chuan Lin Jiaxin -Sun yukui +Sun Yukui diff --git a/people/LiuYinsi/credit/2020-09-25.md b/people/LiuYinsi/credit/2020-09-25.md index c7fb87e..f7652f6 100644 --- a/people/LiuYinsi/credit/2020-09-25.md +++ b/people/LiuYinsi/credit/2020-09-25.md @@ -3,4 +3,3 @@ Yu Chuan Luan Shengde Zhang Dewan Shi Zhichao - diff --git a/people/LuWeitao/credit/2020-09-18.md b/people/LuWeitao/credit/2020-09-18.md index 5f9f6ef..5678b6d 100644 --- a/people/LuWeitao/credit/2020-09-18.md +++ b/people/LuWeitao/credit/2020-09-18.md @@ -1,4 +1,4 @@ -Luan Shengden +Luan Shengde Wu Fengguang Zhang Yuhang Cao Xueliang diff --git a/people/LuWeitao/credit/2020-09-25.md b/people/LuWeitao/credit/2020-09-25.md index decb297..1f2515d 100644 --- a/people/LuWeitao/credit/2020-09-25.md +++ b/people/LuWeitao/credit/2020-09-25.md @@ -2,7 +2,7 @@ Wu Fengguang Du Kaitian Cao Xueliang Zhang Yuhang -Luan Shengden +Luan Shengde Sun Yukui Liu Shaofei Li Yuanchao diff --git a/people/LuanShengde/credit/2020-09-18.md b/people/LuanShengde/credit/2020-09-18.md index 574c682..4a81084 100644 --- a/people/LuanShengde/credit/2020-09-18.md +++ b/people/LuanShengde/credit/2020-09-18.md @@ -3,4 +3,3 @@ Lu Weitao Shi Zhichao Tong Qunfeng Zhang Dewan - diff --git a/people/WangChenglong/credit/2020-09-25.md b/people/WangChenglong/credit/2020-09-25.md index a0e4736..5239ca1 100644 --- a/people/WangChenglong/credit/2020-09-25.md +++ b/people/WangChenglong/credit/2020-09-25.md @@ -3,4 +3,3 @@ Shi Zhichao Wu Zhende Wang Yong Wei Jihui - diff --git a/people/XuXijian/credit/2020-07-31.md b/people/XuXijian/credit/2020-07-31.md index 5cf1aa4..36fea5e 100644 --- a/people/XuXijian/credit/2020-07-31.md +++ b/people/XuXijian/credit/2020-07-31.md @@ -1,4 +1,4 @@ -Zhang Dewang +Zhang Dewan Zhang Yale Yu Chuan Wang Yong diff --git a/people/XuXijian/credit/2020-08-07.md b/people/XuXijian/credit/2020-08-07.md index 056782a..cca2b2f 100644 --- a/people/XuXijian/credit/2020-08-07.md +++ b/people/XuXijian/credit/2020-08-07.md @@ -4,4 +4,4 @@ Li Yuanchao Xiao Shenwei Wei Jihui Yu Chuan -Zhang Dewang +Zhang Dewan diff --git a/people/XuXijian/credit/2020-08-14.md b/people/XuXijian/credit/2020-08-14.md index 7c60171..e9cfd69 100644 --- a/people/XuXijian/credit/2020-08-14.md +++ b/people/XuXijian/credit/2020-08-14.md @@ -1,4 +1,4 @@ -Shi zhichao +Shi Zhichao Luan Shengde Wang Yong Wu Fengguang -- 2.23.0

1 0