- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] src/extract-stats prefix exception handing
by Lu Weitao 28 Sep '20

28 Sep '20

background: extract-stats process is at risk of be interrupted, due to other dependent services(task-queue, es_client) may not working properly solution: 1. use begin ... rescue to handle possible exceptions 2. skip and remove the task from task-queue if the job data does not have post-processing conditions 3. refactor for stats_worker.cr Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 60 ++++++++++++++++++------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..edebf27 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -18,40 +18,50 @@ class StatsWorker def consume_sched_queue(queue_path : String) loop do begin - response = @tq.consume_task(queue_path) + job_id = get_job_id(queue_path).to_s + if job_id # consume the job by post-processing + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + end rescue e STDERR.puts e.message - # incase of many error message when task-queue is busy + # incase of many error message when task-queue, ES does not work sleep(10) - next end + end + end - if response[0] == 200 - job_id = JSON.parse(response[1].to_json)["id"] - job = @es.get_job_content(job_id.to_s) - result_root = job["result_root"] - begin - # extract stats.json - system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" - # storage stats to job in es - store_stats_es(result_root.to_s, job_id.to_s) if result_root - # send mail to submitter for job results - system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" - rescue e - STDERR.puts e.message - next - end - - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - else - sleep(60) - end + # get job_id from task-queue + def get_job_id(queue_path : String) + response = @tq.consume_task(queue_path) + if response[0] == 200 + JSON.parse(response[1].to_json)["id"] + else + # will sleep 60s if no task in task-queue + sleep(60) + nil end end - def store_stats_es(result_root : String, job_id : String) + def result_post_processing(job_id : String, result_root : String, queue_path : String) + return nil unless result_root + + # extract stats.json + system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" + # storage stats to job in es + store_stats_es(result_root, job_id, queue_path) + # send mail to submitter for job results + system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" + end + + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - raise "#{stats_path} file not exists." unless File.exists?(stats_path) + unless File.exists?(stats_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + raise "#{stats_path} file not exists." + end stats = File.open(stats_path) do |file| JSON.parse(file) -- 2.23.0

1 0

[PATCH compass-ci] feat: scheduler: add optional fields into job
by Ren Wen 28 Sep '20

28 Sep '20

Only `multi-qemu` job needs `FLUENTD_SERVER_*`, so move them to the place where calls this function. When calls this function, an array can be passed, like: ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}",] these elements will add into job as job fields. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/jobfile_operate.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index df5af1b..0533346 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -200,14 +200,13 @@ module Jobfile::Operate return target_path end - def self.auto_submit_job(job_file, override_parameter) + def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) job_fields = ["SCHED_HOST=#{ENV["SCHED_HOST"]}", "SCHED_PORT=#{ENV["SCHED_PORT"]}", - "FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", - "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}", "GIT_SERVER=#{ENV["GIT_SERVER"]}", "lab=#{ENV["lab"]}"] + job_fields.concat(other_parameters) if other_parameters cmd = "#{ENV["LKP_SRC"]}/sbin/submit " cmd += job_fields.join(" ") cmd += " -s '#{override_parameter}' #{job_file}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow: container: fix git-mirror start error
by Liu Yinsi 28 Sep '20

28 Sep '20

1. git clone upstream-repo and export REPO_SRC to fix error: No such file or directory @ rb_sysopen - /c/upstream-repos 2. chown lkp:committer /srv/git to fix error: Permission denied @ dir_s_mkdir - /srv/git/b Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/3-code/dev-env | 1 + sparrow/3-code/git | 4 ++++ 3 files changed, 6 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 35456a8..f5c1646 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -6,6 +6,7 @@ chown -R lkp:lkp /srv/es chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result chmod 775 /srv/es/logging-es diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..ae267bd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -27,6 +27,7 @@ EOF cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci +export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF diff --git a/sparrow/3-code/git b/sparrow/3-code/git index b1d2c03..5588696 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -16,3 +16,7 @@ cd /c || exit [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git } + +[ -d "upstream-repos/.git" ] || { + git clone https://gitee.com/wu_fengguang/upstream-repos.git +} -- 2.23.0

1 0

[PATCH compass-ci] Makefile: add ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 Makefile diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..c7505e8 --- /dev/null +++ b/Makefile @@ -0,0 +1,2 @@ +ctags: + ctags -R --links=no --langmap=ruby:.rb.cr -- 2.23.0

1 1

[PATCH compass-ci] fix: system rebooting when executed 2-network/br0
by Xu Xijian 28 Sep '20

28 Sep '20

When 2-network/br0 executed, program will hang up fo a while and then go rebooting. The cause is iptables need time to take effect, echo 1 to /proc/sys/net/ipv4/ip_forward to make it taking effect immediately. Logs(openeuler 20.03): [ 161.565451] + 2-network/iptables [ 161.565453] [ 161.568088] ++ ip route get 1.2.3.4 [ 161.568090] ... [ 168.107379] LKP: rebooting Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- sparrow/2-network/iptables | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..0bf1235 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -11,6 +11,8 @@ BR0_SUBNET=172.18.0.0/16 iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE +# To make upon rules taking effect immediately +echo 1 > /proc/sys/net/ipv4/ip_forward systemctl status firewalld | grep -q "running" && { -- 2.23.0

2 1

[PATCH compass-ci] code-spec: refactor friendly principles
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index e783a0c..f805aec 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -7,6 +7,23 @@ code注意事项 - 一个函数, 不要超过10行代码 - 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 +重构友好编码 +============ + +基本原则: +- 预防自己的代码造成他人的几大困难 + - 难以理解不敢下手 + - 不知道怎么找到*所有*关联逻辑 + - 不知道自己不知道 (unknown unknown) +- 设想将来做重构, 或者需要理解任何一个高层次的功能/流程, + 一个新人应当可以通过 git grep 一两个关键字或者pattern, + 就能找到所有相关底层代码与文档, 而不必担心有所遗漏. + +基本方法: +- 隐式场景显式化: 加注释明确代码的各类隐含目的, 意义, 场景, 数据示例等等 +- 隐式依赖显式化: 定义相同的ID, 把所有的关联文档与各处代码串起来 +- 分散依赖集中化: 定义类/函数/assert, 封装/强制各类设计中的假设, 约定与规则 + PATCH注意事项 ============= -- 2.23.0

1 0

[PATCH lkp-tests] Makefile: support ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Makefile b/Makefile index c40d914b..37387809 100644 --- a/Makefile +++ b/Makefile @@ -23,3 +23,6 @@ doc: tests/%.md: tests/%.yaml lkp gen-doc $< + +ctags: + ctags -R --links=no -- 2.23.0

1 0

[PATCH compass-ci] doc: add code-spec.md
by Wu Fengguang 27 Sep '20

27 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 42 ++++++++++++++++++++++++++++++++++++++ doc/develop-environment.md | 1 + 2 files changed, 43 insertions(+) create mode 100644 doc/code-spec.md diff --git a/doc/code-spec.md b/doc/code-spec.md new file mode 100644 index 0000000..e783a0c --- /dev/null +++ b/doc/code-spec.md @@ -0,0 +1,42 @@ +code注意事项 +============ + +- 代码应当放进函数里不要在脚本顶层直接写逻辑 +- 代码应当放进lib/里脚本本身最小化, 仅解析命令行参数, 调用lib/入口函数 +- 如果出现了全局变量一般意味着需要写一个 class/module 来放这个状态量了 +- 一个函数, 不要超过10行代码 +- 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 + +PATCH注意事项 +============= + +- PATCH title 应当简要说明做了什么 (what) +- PATCH changelog / code comment 应当解释读者无法从代码简单推断的 + - 背景 + - 目的 + - 效果 + - 设计选项和权衡 + - 实际例子 + - 改进数字 + - bug场景, 出错消息原文 +- PATCH 应当加上作者签名 (Signed-off-by: Name <email>) +- PATCH 最好不超过100行, 一个PATCH只做一件事, 避免混杂多个逻辑变更 + +PATCH参考材料 +============= + +The patch subject and changelog should be well written according to suggestions +here: + + https://github.com/thoughtbot/dotfiles/blob/master/gitmessage + https://www.cnblogs.com/cpselvis/p/6423874.html + + http://www.ozlabs.org/~akpm/stuff/tpp.txt + 2: Subject: + 3: Attribution + 4: Changelog + + https://www.kernel.org/doc/html/latest/translations/zh_CN/process/submittin… + 2) 描述你的改动 + 3) 拆分你的改动 + 8）回复评审意见 diff --git a/doc/develop-environment.md b/doc/develop-environment.md index 3af44ac..8760878 100644 --- a/doc/develop-environment.md +++ b/doc/develop-environment.md @@ -50,6 +50,7 @@ Then read through documents lkp-tests/doc/INSTALL.md compass-ci/doc/INSTALL.md + compass-ci/doc/code-spec.md compass-ci/doc/learning-resources.md # crystal compiler -- 2.23.0

1 0

[PATCH todo] credit: fix typos
by Wu Fengguang 26 Sep '20

26 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- people/LiangTao/credit/2020-09-11.md | 2 +- people/LiuShaofei/credit/2020-08-14.md | 2 +- people/LiuYinsi/credit/2020-09-25.md | 1 - people/LuWeitao/credit/2020-09-18.md | 2 +- people/LuWeitao/credit/2020-09-25.md | 2 +- people/LuanShengde/credit/2020-09-18.md | 1 - people/WangChenglong/credit/2020-09-25.md | 1 - people/XuXijian/credit/2020-07-31.md | 2 +- people/XuXijian/credit/2020-08-07.md | 2 +- people/XuXijian/credit/2020-08-14.md | 2 +- 10 files changed, 7 insertions(+), 10 deletions(-) diff --git a/people/LiangTao/credit/2020-09-11.md b/people/LiangTao/credit/2020-09-11.md index 87afa33..464173b 100644 --- a/people/LiangTao/credit/2020-09-11.md +++ b/people/LiangTao/credit/2020-09-11.md @@ -1,4 +1,4 @@ Shi Zhichao Xiao Shenwei Wang Chenglong -Wei Jihiu +Wei Jihui diff --git a/people/LiuShaofei/credit/2020-08-14.md b/people/LiuShaofei/credit/2020-08-14.md index 66d1d24..a5a3f35 100644 --- a/people/LiuShaofei/credit/2020-08-14.md +++ b/people/LiuShaofei/credit/2020-08-14.md @@ -4,4 +4,4 @@ Wang Yong Bai Jing Yu Chuan Lin Jiaxin -Sun yukui +Sun Yukui diff --git a/people/LiuYinsi/credit/2020-09-25.md b/people/LiuYinsi/credit/2020-09-25.md index c7fb87e..f7652f6 100644 --- a/people/LiuYinsi/credit/2020-09-25.md +++ b/people/LiuYinsi/credit/2020-09-25.md @@ -3,4 +3,3 @@ Yu Chuan Luan Shengde Zhang Dewan Shi Zhichao - diff --git a/people/LuWeitao/credit/2020-09-18.md b/people/LuWeitao/credit/2020-09-18.md index 5f9f6ef..5678b6d 100644 --- a/people/LuWeitao/credit/2020-09-18.md +++ b/people/LuWeitao/credit/2020-09-18.md @@ -1,4 +1,4 @@ -Luan Shengden +Luan Shengde Wu Fengguang Zhang Yuhang Cao Xueliang diff --git a/people/LuWeitao/credit/2020-09-25.md b/people/LuWeitao/credit/2020-09-25.md index decb297..1f2515d 100644 --- a/people/LuWeitao/credit/2020-09-25.md +++ b/people/LuWeitao/credit/2020-09-25.md @@ -2,7 +2,7 @@ Wu Fengguang Du Kaitian Cao Xueliang Zhang Yuhang -Luan Shengden +Luan Shengde Sun Yukui Liu Shaofei Li Yuanchao diff --git a/people/LuanShengde/credit/2020-09-18.md b/people/LuanShengde/credit/2020-09-18.md index 574c682..4a81084 100644 --- a/people/LuanShengde/credit/2020-09-18.md +++ b/people/LuanShengde/credit/2020-09-18.md @@ -3,4 +3,3 @@ Lu Weitao Shi Zhichao Tong Qunfeng Zhang Dewan - diff --git a/people/WangChenglong/credit/2020-09-25.md b/people/WangChenglong/credit/2020-09-25.md index a0e4736..5239ca1 100644 --- a/people/WangChenglong/credit/2020-09-25.md +++ b/people/WangChenglong/credit/2020-09-25.md @@ -3,4 +3,3 @@ Shi Zhichao Wu Zhende Wang Yong Wei Jihui - diff --git a/people/XuXijian/credit/2020-07-31.md b/people/XuXijian/credit/2020-07-31.md index 5cf1aa4..36fea5e 100644 --- a/people/XuXijian/credit/2020-07-31.md +++ b/people/XuXijian/credit/2020-07-31.md @@ -1,4 +1,4 @@ -Zhang Dewang +Zhang Dewan Zhang Yale Yu Chuan Wang Yong diff --git a/people/XuXijian/credit/2020-08-07.md b/people/XuXijian/credit/2020-08-07.md index 056782a..cca2b2f 100644 --- a/people/XuXijian/credit/2020-08-07.md +++ b/people/XuXijian/credit/2020-08-07.md @@ -4,4 +4,4 @@ Li Yuanchao Xiao Shenwei Wei Jihui Yu Chuan -Zhang Dewang +Zhang Dewan diff --git a/people/XuXijian/credit/2020-08-14.md b/people/XuXijian/credit/2020-08-14.md index 7c60171..e9cfd69 100644 --- a/people/XuXijian/credit/2020-08-14.md +++ b/people/XuXijian/credit/2020-08-14.md @@ -1,4 +1,4 @@ -Shi zhichao +Shi Zhichao Luan Shengde Wang Yong Wu Fengguang -- 2.23.0

1 0

答复: Welcome to the "Compass-ci" mailing list
by Dukaitian (Dukaitian, Intelligent Computing R&D) 23 Sep '20

23 Sep '20

各位老师、开发者好：由于crystal 被其它公司注册了，及后缀CI的通用性，影响到crystal-ci 的名字，因此需要将项目名称 crystal-ci 修改为 compass-ci 后续我们需要通过compass-ci(a)openeuler.org 邮件列表来继续交流，crystal-ci(a)openEuler.org 预计在一个月后进行停用。由于变更给大家带来的麻烦深表歉意，敬请谅解，感谢大家一如既往的关注及支持。祝好 Jimmy_hero -----邮件原件----- 发件人: compass-ci-request(a)openeuler.org [mailto:compass-ci-request@openeuler.org] 发送时间: 2020年9月23日 17:25 收件人: Dukaitian (Dukaitian, Intelligent Computing R&D) <dukaitian(a)huawei.com> 主题: Welcome to the "Compass-ci" mailing list Welcome to the "Compass-ci" mailing list! To post to this list, send your email to: compass-ci(a)openeuler.org You can unsubscribe or make adjustments to your options via email by sending a message to: compass-ci-request(a)openeuler.org with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI.

1 0