- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] sbin/result2stats: support call function from stats/script
by Lu Weitao 05 Mar '21

05 Mar '21

[Why] old logical: lkp-tests/stats/srcipt < script_log > tmpfile lkp-tests/sbin/dump-stat < tmpfile > $script.json this flow will escape special characters [How] use return of function to keep original characters lkp-tests/stats/srcipt.rb func(read(script_log)) --> hash compass-ci/lib/dump-stats.rb hash --> dump_stats() --> $script.json Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats.rb | 19 ++++++++++++++----- lib/stats_wrapper.rb | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 5 deletions(-) diff --git a/lib/stats.rb b/lib/stats.rb index c01d38d..94e370a 100644 --- a/lib/stats.rb +++ b/lib/stats.rb @@ -14,11 +14,16 @@ end def available_stats vailable_stats = [] + available_stats_func = [] Dir.open(LKP_SRC + '/stats').each do |file| - vailable_stats << file unless file.start_with?('.') + if file.end_with?('.rb') + available_stats_func << file.chomp('.rb') + else + vailable_stats << file unless file.start_with?('.') + end end - vailable_stats + [vailable_stats, available_stats_func] end def assign_default_stats(stats_list) @@ -32,13 +37,17 @@ class Stats def initialize(result_root) @result_root = result_root @job = load_yaml(File.join(result_root, 'job.yaml')) - @available_stats = available_stats + @available_stats, @available_stats_func = available_stats end def extract_stats stats_list = assign_stats_list stats_list.each do |stat| - StatsWrapper.wrapper(*stat) + if @available_stats_func.include?(stat) + StatsWrapper.wrapper_func(*stat) + else + StatsWrapper.wrapper(*stat) + end end end @@ -48,7 +57,7 @@ class Stats stats_list << ['time', @job['suite'] + '.time'] stats_list << 'stderr' @job.each_key do |k| - if @available_stats.include?(k) + if @available_stats.include?(k) || @available_stats_func.include?(k) stats_list << k end end diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 8a6d6a9..09662be 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -6,6 +6,11 @@ LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' require "#{LKP_SRC}/lib/log.rb" require 'tempfile' require 'English' +require_relative './dump_stat' + +Dir[File.expand_path("#{LKP_SRC}/stats/*.rb")].uniq.each do |file| + require file +end PROGRAM_DIR = "#{LKP_SRC}/stats" @@ -35,6 +40,26 @@ module StatsWrapper delete_log_package end + def self.wrapper_func(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless File.exist?("#{(a)log}.yaml") || pretreatment + + if File.exist?("#{(a)log}.yaml") + stat_result = YAML.safe_load("#{(a)log}.yaml") + else + log_lines = read_log(@log) + call_func_cmd = "#{(a)program.gsub('-', '_')}(log_lines)" # eg: proc_vmstats(log_lines) + stat_result = eval(call_func_cmd) + end + return unless DumpStat.dump_stat(@stats_group, stat_result) + + check_empty_json + delete_log_package + end + def self.pretreatment return unless available_program? return unless unzip_log @@ -261,3 +286,11 @@ module StatsWrapper File.delete(@tmpfile) if File.exist?(@tmpfile) end end + +# read line of log +# return Array(String) +def read_log(log_path) + return nil unless File.exist?(log_path) + + File.readlines(log_path) +end -- 2.23.0

2 2

[PATCH v2 compass-ci] src/lib/job.cr: delete os_mount in common_dir for upload rpm
by Li Ping 04 Mar '21

04 Mar '21

[why] rpm has no much to do with os_mount. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..a30c969 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -274,7 +274,11 @@ class Job tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

2 2

[PATCH compass-ci] lib/git_mirror.rb: Add new_refs_log for kibana
by Li Yuanchao 04 Mar '21

04 Mar '21

The logs can be shown as chart by kibana. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bb0c9f9..92a3830 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -192,6 +192,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) + new_refs_log(git_repo, new_refs) end def do_push(fork_key) @@ -509,4 +510,15 @@ class MirrorMain traverse_repodir(repodir, belong) end end + + def new_refs_log(git_repo, new_refs) + log = { + msg: 'new refs', + date: Date.today, + time: Time.now, + repo: git_repo, + nr_new_branch: new_refs[:heads].length + }.to_json + puts log + end end -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit
by Wang Chenglong 04 Mar '21

04 Mar '21

[Why] In 'job.yaml The 'os_version' should use quotation marks around the value, converted to a string type. Otherwise we use 'os_version: 6.10', The submit will get error. wcl@crystal ~/my_yaml% cat iperf.yaml os: centos os_arch: aarch64 os_version: 6.10 os_mount: cifs wcl@crystal ~/my_yaml% submit iperf.yaml submit_id=983c3605-03e2-4e8f-8ac4-c78325463fde submit iperf.yaml failed, got job id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/6.1/boot/vmlinuz': No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- sbin/submit | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/submit b/sbin/submit index a6f0cd387..993e27f96 100755 --- a/sbin/submit +++ b/sbin/submit @@ -147,6 +147,10 @@ ARGV.each do |jobfile| jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') jobs['node_roles'] ||= 'server client' if jobs['cluster'] jobs['submit_id'] = submit_id + jobs['os_version'].is_a?(String) || ( + puts "Please use quotation marks around value: os_version: \'#{jobs['os_version']}\'" + exit + ) jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] raise 'Please configure SCHED_PORT' unless job['SCHED_PORT'] -- 2.23.0

2 1

help star
by Wu Fengguang 04 Mar '21

04 Mar '21

Jihui is our help star, thanks Jihui! ~/todo/bin% ./help-star 21.6 Wu Fengguang 13.2 Wei Jihui 12.2 Cao Xueliang 11.0 Wu Zhende 5.8 Lin Jiaxin 5.6 Luan Shengde 5.3 Liu Yinsi 5.1 Lu Weitao 4.5 Yu Chuan 4.5 Wang Yong 3.7 Li Ping 2.5 Zhang Dewan 2.2 Cui Yili 1.8 Wang Chenglong 1.1 Bai Jing 0.0 Tong Qunfeng 0.0 Xu Xijian 0.0 Hu Xuejiao 0.0 Du Kaitian 0.0 Liu Shaofei 0.0 Xiao Shenwei 0.0 Zhang Yale 0.0 Li Yuanchao 0.0 Ren Wen 0.0 Wang Shuai 0.0 Lu Kaiyi 0.0 Cui Xucui

1 0

git count
by Wu Fengguang 04 Mar '21

04 Mar '21

Xueliang, Zhende is our code star, thanks Xueliang, Zhende! wfg@crystal ~/todo/bin% ./git-count 8 Cao Xueliang 8 Wu Zhende 6 Liu Yinsi 6 Wei Jihui 4 Yu Chuan 4 Wang Yong 4 Li Yuanchao 3 Lu Weitao 2 Wang Chenglong 2 Luan Shengde 2 Lin Jiaxin 0 Bai Jing 0 Tong Qunfeng 0 Xu Xijian 0 Hu Xuejiao 0 Du Kaitian 0 Liu Shaofei 0 Sun Yukui 0 Wu Fengguang 0 Xiao Shenwei 0 Zhang Yale 0 Zhang Yu 0 Cui Yili 0 Ren Wen 0 Zhang Dewan 0 Li Ping 0 Lu Kaiyi 0 Shi Zhichao w

1 0

review star
by Wu Fengguang 04 Mar '21

04 Mar '21

Jihui is our review star again, thanks Jihui! wfg@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 41 22 45 20 Wu Fengguang 61 18 27 15 Wei Jihui 68 9 14 12 Cao Xueliang 41 14 36 10 Luan Shengde 23 0 0 3 Yu Chuan 13 2 2 2 Wang Yong 15 1 2 2 Wang Chenglong 2 1 0 1 Bai Jing 9 0 0 0 Wu Zhende 30 0 0 0 Liu Yinsi 13 0 0 0 Li Yuanchao 13 0 0 0 Lu Weitao 14 0 0 0 Xu Xijian 12 0 0 0 Li Ping 5 1 1 0 Lin Jiaxin 13 0 0 -1 Cui Yili

1 0

[PATCH v3 compass-ci] container/submit: add attach lkp-tests
by Luan Shengde 04 Mar '21

04 Mar '21

add attach lkp-tests to the container when run the container to submit jobs. [why] enable user edit code or job file(s). user can edit the code or job file(s) according to their requires. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 1 + 1 file changed, 1 insertion(+) diff --git a/container/submit/submit b/container/submit/submit index 6236892..ada05f9 100755 --- a/container/submit/submit +++ b/container/submit/submit @@ -13,6 +13,7 @@ cmd=( --name=submit-$USER-$data_suffix -it -v /etc/compass-ci:/etc/compass-ci:ro + -v $LKP_SRC:/root/lkp-tests:ro -v $HOME/.config:/root/.config:ro -v $HOME/.ssh:/root/.ssh:rw submit -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: improve bisect report content
by Cao Xueliang 04 Mar '21

04 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 7ce5660..782c702 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -54,7 +54,7 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, - We found some error/warning(s) and the first bad commit in the below project: + Some error/warning(s) are found in git url: #{@upstream_url}/#{@commit_id} git commit: #{@commit_id} ("#{(a)git_commit.subject}") -- 2.23.0

1 0

[PATCH compass-ci] sparrow/dev-env: expose the SRV_HTTP host and port for job
by Cao Xueliang 04 Mar '21

04 Mar '21

Reference: 36a7848fe733b36a5b75c81560b32b400c4c1b98 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index df7b1ce..403ccb1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -84,6 +84,12 @@ LKP_SERVER: GIT_SERVER: GITCACHE_HOST: GITCACHE_PORT: +SRV_HTTP_RESULT_HOST: +SRV_HTTP_RESULT_PORT: +SRV_HTTP_OS_HOST: +SRV_HTTP_OS_PORT: +SRV_HTTP_GIT_HOST: +SRV_HTTP_GIT_PORT: SRV_HTTP_CCI_HOST: SRV_HTTP_CCI_PORT: SEND_MAIL_HOST: -- 2.23.0

1 0