- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] bin/lkp-setup-rootfs: modify test-machine is not restarted
by Hu Xuejiao 11 Dec '20

11 Dec '20

[why] When force_reboot is not added to the job.yaml, the test-machine is "no KERNEL found". Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/lkp-setup-rootfs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 0eabea75..cca1b53b 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -61,7 +61,10 @@ job_does_not_need_reboot() job_force_reboot() { - grep -q "^force_reboot: \+1$" $job + grep -q "^force_reboot: 1$" $job && return 0 + grep -q "^do_not_reboot: 1$" $job || return 0 + + return 1 } reboot_for_next_job() -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix syntax errors caused by missing double quotes
by Lin Jiaxin 11 Dec '20

11 Dec '20

error: PKGBUILD.src: line 2: syntax error near unexpected token `&' PKGBUILD.src: line 2: `glassfish-5.1.0.zip::http://www.eclipse.org/downloads/download.php?file=/glassfish/glassfish-5.1.0.zip&r=1' PKGBUILD.src: line 3: glassfish5.service: command not found PKGBUILD.src: line 4: syntax error near unexpected token `)' PKGBUILD.src: line 4: `)' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 8a138055..0b2794b1 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -112,7 +112,7 @@ replace_source() } url=$(echo "$url" | sed 's|https://|http://|g') - echo "$url" >> $1 + echo "\"$url\"" >> $1 done echo ")" >> $1 } -- 2.23.0

3 3

[PATCH v2 lkp-tests] tests/build-pkg: fix syntax errors caused by missing quotes
by Lin Jiaxin 11 Dec '20

11 Dec '20

error: PKGBUILD.src: line 2: syntax error near unexpected token `&' PKGBUILD.src: line 2: `glassfish-5.1.0.zip::http://www.eclipse.org/downloads/download.php?file=/glassfish/glassfish-5.1.0.zip&r=1' PKGBUILD.src: line 3: glassfish5.service: command not found PKGBUILD.src: line 4: syntax error near unexpected token `)' PKGBUILD.src: line 4: `)' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 8a138055..aeaf1778 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -112,7 +112,7 @@ replace_source() } url=$(echo "$url" | sed 's|https://|http://|g') - echo "$url" >> $1 + echo \'$url\' >> $1 done echo ")" >> $1 } -- 2.23.0

1 0

[PATCH compass-ci 2/2] add options for compare_by_template
by Lu Kaiyi 11 Dec '20

11 Dec '20

if theme is json like below: compare -t compare_template.yaml --theme json display json format output result, otherwise, display table format output result by default. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 11 ++++++++--- sbin/compare | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index f7bb56a..db102f1 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -507,11 +507,16 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) + theme = options[:theme] formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_data = FormatTableData.new(echart_results) - table_data.show_table + if theme == 'json' + print JSON.pretty_generate(echart_results) + else + table_data = FormatTableData.new(echart_results) + table_data.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: friendly show compare template result in command line
by Lu Kaiyi 11 Dec '20

11 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the json format result of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [usage] compare -t <compare_template.yaml> [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | openeuler 20.03 standard_deviation | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | centos 7.6 average | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | centos 7.6 standard_deviation | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 53 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 ++- 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..aa5179f 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +67,58 @@ class FormatEchartData end end +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_field_names(table_title, table) + set_table_title + add_row(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = @title + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_row(table) + row_names = %w[average standard_deviation] + row_names.each do |row_name| + row = table[row_name]['source'][1] + row_title = [row[0], row_name].join(' ') + format_data_row = row[1..-1] + format_data_row.map! { |data| format('%.2f', data) } + @tb.add_row([row_title, *format_data_row]) + end + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5293989..f7bb56a 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -510,7 +510,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_data = FormatTableData.new(echart_results) + table_data.show_table end # Format Fields -- 2.23.0

1 0

help star
by Wu Fengguang 11 Dec '20

11 Dec '20

Xueliang is our help star, thanks Xueliang! wfg@crystal ~/todo% bin/help-star people/*/credit/2020-12-0[34].md 22.7 Cao Xueliang 20.0 Wei Jihui 19.8 Wu Zhende 16.0 Luan Shengde 13.6 Wu Fengguang 13.3 Lu Weitao 11.9 Xiao Shenwei 11.0 Bai Jing 9.3 Yu Chuan 7.7 Zhang Yu 7.4 Ren Wen 6.7 Zhang Yale 6.7 Cui Yili 6.2 Xu Xijian 6.2 Lin Jiaxin 5.7 Liu Yinsi 5.6 Zhang Dewan 5.6 Wang Yong 5.1 Lu Kaiyi 4.4 Li Ping 4.2 Hu Xuejiao 3.8 Liu Shaofei 3.5 Du Kaitian 2.6 Sun Yukui 0.9 Li Yuanchao 0.0 Tong Qunfeng 0.0 Wang Chenglong 0.0 Wang Shuai 0.0 Cui Xucui

1 0

review star
by Wu Fengguang 11 Dec '20

11 Dec '20

Shenwei is our review star, thanks Shenwei! wfg@crystal ~/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 19 12 34 21 Xiao Shenwei 95 30 44 19 Wei Jihui 28 16 25 17 Liu Shaofei 79 13 16 16 Cao Xueliang 20 4 18 13 Wang Yong 41 16 36 10 Wu Fengguang 34 9 10 9 Zhang Yu 22 7 18 9 Luan Shengde 30 4 17 8 Yu Chuan 26 3 8 7 Sun Yukui 19 5 7 6 Wu Zhende 18 4 4 5 Cui Yili 8 8 8 4 Li Ping 33 6 5 4 Hu Xuejiao 63 7 11 4 Wang Chenglong 19 7 15 4 Lu Kaiyi 6 2 4 3 Xu Xijian 13 1 2 2 Lu Weitao 14 4 5 2 Lin Jiaxin 17 5 5 2 Liu Yinsi 15 1 1 1 Bai Jing 4 0 0 0 Li Yuanchao 33 2 0 0 Zhang Yale 2 0 0 0 Zhang Dewan 20 0 0 0 Ren Wen

1 0

git count
by Wu Fengguang 11 Dec '20

11 Dec '20

Zhende is our code star again, thanks Zhende! wfg@crystal ~/todo% bin/git-count 13 Wu Fengguang 12 Wu Zhende 9 Cao Xueliang 9 Zhang Yale 7 Hu Xuejiao 6 Bai Jing 6 Liu Shaofei 6 Lu Weitao 6 Liu Yinsi 6 Wei Jihui 6 Zhang Yu 6 Luan Shengde 5 Yu Chuan 5 Cui Yili 5 Wang Chenglong 5 Li Yuanchao 5 Lu Kaiyi 4 Wang Yong 4 Lin Jiaxin 3 Sun Yukui 3 Ren Wen 2 Xu Xijian 2 Xiao Shenwei 1 Li Ping 0 Tong Qunfeng 0 Du Kaitian 0 Zhang Dewan

1 0

[PATCH compass-ci] logging: fix time difference of eight hours
by Wu Zhende 11 Dec '20

11 Dec '20

If the time zone information is not contained, the ES saves time as the UTC time by default. As a result, the kibana display time is eight hours later. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 2 +- src/lib/json_logger.cr | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index b3daeb4..7bf2229 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -11,7 +11,7 @@ @type record_transformer enable_ruby <record> - time ${time.strftime('%Y-%m-%dT%H:%M:%S')} + time ${time.strftime('%Y-%m-%dT%H:%M:%S.%3N+0800')} </record> </filter> diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index dd87020..6d61197 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -18,7 +18,7 @@ class JSONLogger < Logger Logger::Formatter.new do | severity, datetime, progname, msg, io| get_env_info(@env.as(HTTP::Server::Context)) if @env level_num = severity.to_i32 - datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S") + datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S.%3N+0800") logger_hash = JSON.parse(%({"level_num": #{level_num}, "level": "#{severity}", "time": "#{datetime}" -- 2.23.0

2 1

[PATCH lkp-tests] fix: depends/lkp: add ntpdate server
by Wang Chenglong 11 Dec '20

11 Dec '20

[Why] In $LKP_SRC/lib/bootstrap.s use ntpdate command to synchronize time. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- distro/depends/lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/lkp b/distro/depends/lkp index 6c8b6bf2..750b5c59 100644 --- a/distro/depends/lkp +++ b/distro/depends/lkp @@ -4,3 +4,4 @@ time kmod gzip which +ntpdate -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler: subqueue: consume subqueue
by Ren Wen 11 Dec '20

11 Dec '20

Find subqueues related to the host default queues of all users, and consume the job from random one. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 20 +++++++++++++++++++- src/scheduler/redis_client.cr | 4 ++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 6f51640..77071db 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -56,7 +56,25 @@ class Sched queues << item.strip end - return rand_queues(queues) + user_queues = [] of String + queues.each do |queue| + matched_queues = @redis.keys("*/sched/#{queue}*") + matched_queues.each do |mq| + user_queues << "#{mq}" + end + end + + user_queues = Set.new(user_queues) + user_queues = user_queues.map do |queue| + if queue =~ /sched\/(.+)\/ready$/ + $1 + else + "" + end + end + user_queues.reject!(&.empty?) + + return rand_queues(user_queues) end def get_job_from_queues(queues, testbox) diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index b816d67..1253108 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -16,6 +16,10 @@ class Redis::Client @client = Redis::PooledClient.new(host: host, port: port, pool_size: 25, pool_timeout: 0.01) end + def keys(pattern) + @client.keys(pattern) + end + def hash_set(key : String, field, value) @client.hset(key, field.to_s, value.to_s) end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] scheduler: subqueue: change queue to queue/subqueue
by Ren Wen 11 Dec '20

11 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d071d69..2c33ffd 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,7 +14,7 @@ class Sched end response = @task_queue.hand_over_task( - "sched/#{job.queue}", "extract_stats", job_id + "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] scheduler: subqueue: submit user's job to subqueue
by Ren Wen 11 Dec '20

11 Dec '20

Every user has a subqueue, user's email by default. Queue will be "queue/subqueue" when submitting job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 6 ++++++ src/scheduler/submit_job.cr | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7ded6d8..0083a7d 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -80,6 +80,7 @@ class Job testbox lab queue + subqueue initrd_pkg initrd_deps initrds_uri @@ -161,6 +162,7 @@ class Job set_lkp_server() set_sshr_info() set_queue() + set_subqueue() end private def set_kernel @@ -266,6 +268,10 @@ class Job end end + private def set_subqueue + self["subqueue"] = self["my_email"] + end + # if not assign tbox_group, set it to a match result from testbox # ?if job special testbox, should we just set tbox_group=textbox private def update_tbox_group_from_testbox diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index f93ad2b..3909cb7 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -107,7 +107,7 @@ class Sched # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) - queue = job.queue + queue = "#{job.queue}/#{job.subqueue}" # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? -- 2.23.0

1 0

Re: [PATCH v4 compass-ci] doc/job: add result_root.md
by Zhang Yu 11 Dec '20

11 Dec '20

>>+ result_root: /result/host-info/2020-12-09/vm-2p8g/openeuler-20.03-aarch64/crystal.166995 > > the example missing the field $pp_params. good Thanks, Zhangyu > >>-- >>2.23.0 >>

1 0

[PATCH compass-ci] src/scheduler: fix ameba warning
by Zhang Yu 11 Dec '20

11 Dec '20

Inspecting 1 file. create_job_cpio.cr:83:35 [C] Layout/TrailingWhitespace: Trailing whitespace detected > job_content.delete("job2sh") ^ Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/scheduler/create_job_cpio.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr index cfa40ab..1101a0f 100644 --- a/src/scheduler/create_job_cpio.cr +++ b/src/scheduler/create_job_cpio.cr @@ -80,7 +80,7 @@ class Sched # put job2sh in an array if job_content.has_key?("job2sh") tmp_job_sh_content = job_content["job2sh"] - job_content.delete("job2sh") + job_content.delete("job2sh") job_sh_array = [] of JSON::Any tmp_job_sh_content.as_h.each do |_key, val| -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/job: add result_root.md
by Zhang Yu 11 Dec '20

11 Dec '20

Add a document for introducing the result_root Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/result_root.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/job/result_root.md diff --git a/doc/job/result_root.md b/doc/job/result_root.md new file mode 100644 index 0000000..c2b82c5 --- /dev/null +++ b/doc/job/result_root.md @@ -0,0 +1,10 @@ +# result_root + +Meaning: +- When a test job is executed, some result files will be generated and stored in the "result_root" + directory, you can check the result file in the web page or manually go to the "result_root" + directory for a check +- The default value of this field is: + result_root: /result/$suite/$submit_date/$tbox_group/$rootfs/$pp_params/$id + Here is an example of this field: + result_root: /result/host-info/2020-12-09/vm-2p8g/openeuler-20.03-aarch64/crystal.166995 -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/job: add result_root.md
by Zhang Yu 11 Dec '20

11 Dec '20

Add a document for introducing the result_root Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/result_root.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/job/result_root.md diff --git a/doc/job/result_root.md b/doc/job/result_root.md new file mode 100644 index 0000000..c3e0a01 --- /dev/null +++ b/doc/job/result_root.md @@ -0,0 +1,8 @@ +# result_root + +Meaning: +- When a test job is executed, some result file will generated and stored in the "result_root" + directory, you can check the result file in the web page or manually go to the "result_root" + directory for a check +- The default value of this field is: + result_root: "/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}"/#{pp_params}/#{id}" -- 2.23.0

2 2

[PATCH v2 compass-ci] container: add prompt message
by Bai Jing 11 Dec '20

11 Dec '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- container/conserver/generate_conserver.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/conserver/generate_conserver.rb b/container/conserver/generate_conserver.rb index 4e923c2..a0ee23b 100755 --- a/container/conserver/generate_conserver.rb +++ b/container/conserver/generate_conserver.rb @@ -12,9 +12,12 @@ def generate_conserver(lab) return unless File.exist?('conserver-head.cf') FileUtils.cp('conserver-head.cf', 'conserver.cf') - + host_dir = "#{CCI_REPOS}/lab-#{lab}/hosts/" - return unless Dir.exist?(host_dir) + unless Dir.exist?(host_dir) + puts "please check whether the #{host_dir} directory exists " + return + end Dir.each_child(host_dir) do |host| ipmi_ip = YAML.load_file("#{host_dir}#{host}")['ipmi_ip'] -- 2.23.0

1 0

[PATCH compass-ci] How to add a machine to compass-ci
by Zhang Dewan 10 Dec '20

10 Dec '20

Signed-off-by: Zhang Dewan <513619464(a)qq.com> How a new machine add to compass-ci Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- ...00\350\210\254\350\277\207\347\250\213.md" | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 "doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" diff --git "a/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" new file mode 100644 index 0000000..340ffea --- /dev/null +++ "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" @@ -0,0 +1,54 @@ + +��--�� + +## ��: +- ��compass-ci +- ��scheduler�� + +## �� +### 1. �� +��z9��redis-db. + +x.x.x.x:xxxx> `hgetall "sched/mac2host"` # ��redis, �� + +### 2. �� +��scheduler�� +`curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}"` +`curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host2queues?host=${hostname}&queues=${api_queues}"` + +��mac��hostname��api_queues + +#### 2.1 �� +##### 2.1.1 mac +��mac�� + +###### 2.1.1.1 ��mac��TaiShan�� +- ��bmc��--��--��--��--��--port1�� +- ��bios��Advanced--Network Device List�� +- ��bmc�� +`ipmitool -I lanplus -H $cur_ip -U $iBMC_user -P $iBMC_passwd raw 0x30 0x90 0x01 0x03 0x0${nic_idx} 0x00|awk '{print $5,$6,$7,$8,$9,$10}' |sed 's/\s/\-/g'` +- ��os�� +`ip link | grep "^2:" -A 1| grep link |awk '{print $2}` + +##### 2.1.2 hostname +��mac��hostname�� + +###### 2.1.2.1 ��hostname��Taishan�� +` taishan200-2280-${cpuNum}s${coreNum}p-${memSize:-256}g--${aN}` +> taishan200-2280 �� +> cpuNum cpu�� +> coreNum cpu�� +> memSize �� +> aN �� + +#### 2.1.3 api_queues +��hostname�� +��taishan200-2280-2s64p-256g--a100, ��api_queues��taishan200-2280-2s64p-256g + +## �� +### 1. �� +��pxe��scheduler�� + +### 2. ��PXE�� +- ��pxe�� +- �� -- 2.23.0

2 1

[PATCH lkp-tests] bisect: bisect test use top level key
by Cao Xueliang 10 Dec '20

10 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- jobs/bisect.yaml | 2 -- tests/bisect | 6 +++--- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/jobs/bisect.yaml b/jobs/bisect.yaml index 8b0001df..5fab30bf 100644 --- a/jobs/bisect.yaml +++ b/jobs/bisect.yaml @@ -2,5 +2,3 @@ suite: bisect docker_image: delimiter:latest testbox: dc-1g--delimiter bisect: - job_id: - error_id: diff --git a/tests/bisect b/tests/bisect index ab280b95..33303296 100755 --- a/tests/bisect +++ b/tests/bisect @@ -1,13 +1,13 @@ #!/usr/bin/env ruby -# - job_id -# - error_id +# bad_job_id +# error_id # SPDX-License-Identifier: MulanPSL-2.0+ # frozen_string_literal: true require_relative "#{ENV['CCI_SRC']}/lib/git_bisect" require_relative "#{ENV['CCI_SRC']}/lib/mail_bisect_result" -task = {'job_id' => ENV['job_id'], 'error_id' => ENV['error_id']} +task = {'bad_job_id' => ENV['bad_job_id'], 'error_id' => ENV['error_id']} gb = GitBisect.new(task) result = gb.find_first_bad_commit -- 2.23.0

1 0

[PATCH compass-ci] container: add prompt message
by Bai Jing 10 Dec '20

10 Dec '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- container/conserver/generate_conserver.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/conserver/generate_conserver.rb b/container/conserver/generate_conserver.rb index 4e923c2..8914247 100755 --- a/container/conserver/generate_conserver.rb +++ b/container/conserver/generate_conserver.rb @@ -12,9 +12,12 @@ def generate_conserver(lab) return unless File.exist?('conserver-head.cf') FileUtils.cp('conserver-head.cf', 'conserver.cf') - + host_dir = "#{CCI_REPOS}/lab-#{lab}/hosts/" - return unless Dir.exist?(host_dir) + unless Dir.exist?(host_dir) do + puts "please check whether the #{host_dir} directory exists " + return + end Dir.each_child(host_dir) do |host| ipmi_ip = YAML.load_file("#{host_dir}#{host}")['ipmi_ip'] -- 2.23.0

1 0

[PATCH v5 lkp-tests] lkp-tests/tests: add rpmbuild-pkg to build RPM
by Zhang Yale 10 Dec '20

10 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/rpmbuild-pkg | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100755 tests/rpmbuild-pkg diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg new file mode 100755 index 00000000..156ca0f6 --- /dev/null +++ b/tests/rpmbuild-pkg @@ -0,0 +1,59 @@ +#!/bin/bash +# - rpm_repo + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh + +[ -n "$rpm_repo" ] || die "rpm_repo is empty" +benchmark=${rpm_repo##*/} +sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark}" + +init_workspace() +{ + # generate workspace in ${HOME} + rpmdev-setuptree +} + +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" +} + +download_rpm_repo() +{ + filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + for pkgfile in ${filelist[*]} + do + local dir="SOURCES" + echo $pkgfile | egrep "\.spec$" && dir="SPECS" + get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + done +} + +build_rpm() +{ + spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $benchmark.spec` + # Install build depends + yum-builddep -y $spec_dir + # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE + spectool -g -R $spec_dir + # Building rpm or srpm packages + rpmbuild -ba $spec_dir +} + +upload_rpm_pkg() +{ + # rpm package will be generated in "${HOME}/rpmbuild/SRPMS" and "${HOME}/rpmbuild/RPMS" + for file in $(find ${HOME}/rpmbuild/ -type f -name "*.rpm") + do + upload_one_curl ${file} ${sync_dest} + done +} + +init_workspace +download_rpm_repo +build_rpm +upload_rpm_pkg -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] iso2rootfs: allow install some packages to result rootfs
by Yu Chuan 10 Dec '20

10 Dec '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 4 ++++ tests/iso2rootfs | 34 ++++++++++++++++++++++++++++------ 2 files changed, 32 insertions(+), 6 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index c67faf44147d..a84f7583de94 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -34,6 +34,10 @@ iso2rootfs: # - no_fstab: will comment all line in /etc/fstab. config_rootfs: + ## install pkgs for result rootfs + ## - example: vim, git, xxx + rootfs_install_pkgs: + ################# # dailybuild related fields ################# diff --git a/tests/iso2rootfs b/tests/iso2rootfs index cd64044f0833..8625b6484d89 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -203,17 +203,24 @@ get_cgz() tar -xf "${cgz_path_t}" -C "${CGZ_PATH}" } -install_pkgs() +get_pkg_installer() { - local pkg_installer - command -v yum && pkg_installer="yum" + local installer + + has_cmd yum && installer=yum + has_cmd apt && installer=apt - [[ -n "$pkg_installer" ]] || die "cannot find pkg installer." + [ -n "$installer" ] || die "cannot find pkg installer." + echo $installer +} + +install_pkgs() +{ + local installer=$(get_pkg_installer) - local pt for pt do - "$pkg_installer" install -y "$pt" || die "cannot install pkg: $pt." + "$installer" install -y "$pt" || die "cannot install pkg: $pt." done } @@ -367,11 +374,26 @@ disable_fstab() sed -i 's/^$[^#]$/# \1/g' "${ROOTFS_DES_DIR}/etc/fstab" } +install_pkgs_for_rootfs() +{ + [ -n "$rootfs_install_pkgs" ] || return 0 + + local installer=$(get_pkg_installer) + + if [ "$installer" == "yum" ]; then + "$installer" install -y --skip-broken --installroot=${ROOTFS_DES_DIR} ${rootfs_install_pkgs//,/ } + elif [ "$installer" == "apt" ]; then + "$installer" install -y --ignore-missing --fix-missing --fix-broken -o RootDir=${ROOTFS_DES_DIR} ${rootfs_install_pkgs//,/ } + fi +} + config_rootfs() { config_dns_resolver disable_selinux disable_fstab + + install_pkgs_for_rootfs } ############ test rootfs ############ -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] iso2rootfs: add three required fields when submit test job.yaml
by Yu Chuan 10 Dec '20

10 Dec '20

[Why] Since the ability `verify account information` was added on submit job.yaml, so add three required fields when submit test job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 52 ++++++++++++++++++++++++++++++++++---------- tests/iso2rootfs | 14 ++++++++++-- 2 files changed, 52 insertions(+), 14 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 9123bef24ae5..c67faf44147d 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -1,35 +1,63 @@ suite: iso2rootfs category: benchmark iso2rootfs: + ################# # iso related fields to be used to generate rootfs + ################# + iso_os: iso_arch: iso_version: + ################# # place the result rootfs related fields - ## 1. result rootfs will be placed in the following localtion on the - ## remote file server: - ## - {remote_file_server}/{rootfs_path}/{iso_os}/{iso_arch}/ - ## 2. remote file server protocols current supported: - ## - nfs - ## - cifs + ################# + + # 1. Result rootfs will be placed in the following location on the + # remote file server: + # - {remote_file_server}/{rootfs_path}/{iso_os}/{iso_arch}/ + # 2. Remote file server protocols current supported: + # - nfs + # - cifs rootfs_protocol: rootfs_server: rootfs_path: + ################# # config rootfs related fields - ## supported fields: - ## - dns: will config /etc/resolv.conf. - ## - no_selinux: will disable selinux in /etc/selinux/config. - ## - no_fstab: will comment all line in /etc/fstab. + ################# + + # you can config add some configurations of the result rootfs. + # supported fields: + # - dns: will config /etc/resolv.conf. + # - no_selinux: will disable selinux in /etc/selinux/config. + # - no_fstab: will comment all line in /etc/fstab. config_rootfs: + ################# # dailybuild related fields + ################# + + # dailybuild_iso_url_file: + # - The `dailybuild iso url file` content is the url of a iso. + # - The iso checksum file also exists on the network, and the checksum file path is "{dailybuild_iso_url}.check256sum" + # - demo: + # root@localhost ~% curl http://1.1.1.1/dailybuilds/openEuler-20.03-LTS-aarch64-LTS/release_iso + # http://1.1.1.1//dailybuilds/1970-01-01-00-00-00/openEuler-20.03-LTS-aarch64… dailybuild_iso_url_file: + ################# # submit test yaml related fields - ## 1. you can add as many jobs as you like. - ## 2. the following three fields is requied for every test job. + ################# + + # three required fields when submit test yaml + submit_email: + submit_name: + submit_uuid: + + # submit target tests yaml + ## 1. You can add as many jobs as you like. + ## 2. The following three fields is required for every test job. test1_yaml: test1_os_mount: test1_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index ce02080875ce..cd64044f0833 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -448,11 +448,21 @@ EOF ############ submit test job yaml ############ generate_global_yaml() { + [ -n "${submit_email}" ] || die "cannot find value of var: submit_email" + [ -n "${submit_name}" ] || die "cannot find value of var: submit_name" + [ -n "${submit_uuid}" ] || die "cannot find value of var: submit_uuid" + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" mkdir -p "$(dirname "${config_yaml}")" - echo "SCHED_HOST: ${SCHED_HOST}" >> "${config_yaml}" - echo "SCHED_PORT: ${SCHED_PORT}" >> "${config_yaml}" + + cat <<-EOF > "${config_yaml}" + SCHED_HOST: ${SCHED_HOST} + SCHED_PORT: ${SCHED_PORT} + my_email: ${submit_email} + my_name: ${submit_name} + my_uuid: ${submit_uuid} + EOF } generate_submit_append_str() -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] iso2rootfs: result rootfs support disable fstab
by Yu Chuan 10 Dec '20

10 Dec '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 1 + tests/iso2rootfs | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 2c4d3a93ff62..9123bef24ae5 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -21,6 +21,7 @@ iso2rootfs: ## supported fields: ## - dns: will config /etc/resolv.conf. ## - no_selinux: will disable selinux in /etc/selinux/config. + ## - no_fstab: will comment all line in /etc/fstab. config_rootfs: # dailybuild related fields diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 9adbc930b48f..ce02080875ce 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -361,10 +361,17 @@ disable_selinux() sed -i 's/^SELINUX=.*/SELINUX=disabled/g' "${ROOTFS_DES_DIR}/etc/selinux/config" } +disable_fstab() +{ + echo "${config_rootfs}" | grep -qw 'no_fstab' || return 0 + sed -i 's/^$[^#]$/# \1/g' "${ROOTFS_DES_DIR}/etc/fstab" +} + config_rootfs() { config_dns_resolver disable_selinux + disable_fstab } ############ test rootfs ############ -- 2.23.0

1 0