- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/mail_bisect_result: bisect email use new content
by Cao Xueliang 14 Dec '20

14 Dec '20

1. Add software name and first error message in subject 2. Add gcc version, error messages and job url in mail body Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- lib/mail_bisect_result.rb | 20 +++++++++----------- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 189edb8..97ad467 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -95,7 +95,7 @@ class GitBisect error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'error_messages' => error_messages.join("\n")] + 'error_messages' => error_messages] end # first search the good commit in db diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5a208c4..44a5645 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -19,23 +19,21 @@ class MailBisectResult end def compose_mail - subject = "[Compass-CI] #{(a)repo}.git: bisect result" + subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{@error_messages[0]}" + job_url = "job url: http://183.134.196.212:11300/#{ENV['result_root']}\n" ? ENV['result_root'] : '' body = <<~BODY Hi #{(a)git_commit.author_name}, - Bisect completed for + git url: #{(a)git_commit.url} + git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") - url: #{(a)git_commit.url} - - This is a bisect email from compass-ci. We met some problems when test with new commits. - Would you help to check what happend? - After submitting a job we noticed an error response due to the commit: - - commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") + gcc version: 7.3.0 error_messages: - #{@error_messages} + #{(a)error_messages.join("\n")} - https://gitee.com/openeuler/compass-ci + #{job_url} + Regards, + Compass CI team BODY to = 'caoxl(a)crystal.ci' @hash = { 'to' => to, 'body' => body, 'subject' => subject } -- 2.23.0

2 2

[PATCH lkp-tests] sbin/set-env.sh: modify script tightening specification
by Hu Xuejiao 14 Dec '20

14 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/set-env.sh | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/sbin/set-env.sh b/sbin/set-env.sh index 0b197afc..27471bb5 100644 --- a/sbin/set-env.sh +++ b/sbin/set-env.sh @@ -6,10 +6,10 @@ write_shell_profile() { shell_profile_file=$(shell_profile) if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file + echo $@ >> $shell_profile_file else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file + echo "export LKP_SRC=$PWD" >> $shell_profile_file + echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file fi source $shell_profile_file @@ -18,11 +18,11 @@ write_shell_profile() write_host() { if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') fi memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') memory=$(echo $memory_new | awk '{print int($0)+1}')G -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: add configuration files
by Liu Yinsi 14 Dec '20

14 Dec '20

scheduler need to read administrator account configuration info, such as account autosubmit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index b41d55a7c..e6a96fd1f 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -427,6 +427,7 @@ class Job end Dir.glob(["/etc/compass-ci/defaults/*.yaml", + "/etc/compass-ci/account/*.yaml", "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| load_one_defaults(file, @job) end -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/auto-submit: fix failed to submit job in auto-submit container
by Liu Yinsi 14 Dec '20

14 Dec '20

[why] when submit job in container auto-submit, error message: /c/lkp-tests/sbin $ ./submit testbox=vm-2p8g iperf.yaml submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email' because auto-submit not mount directory /etc/compass-ci/account, it can't find autosubmit.yaml [how] mount /etc/compass-ci/account in auto-submit/start Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/start b/container/auto-submit/start index 4632ab0..c25fbdd 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,6 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro + -v /etc/compass-ci/account:/etc/compass-ci/account:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow/3-code: add administrator account autosubmit
by Liu Yinsi 14 Dec '20

14 Dec '20

configure separate administrator account, distinguish the job's committers, can be easily data statistics. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index fb16000..69b389f 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -41,6 +41,13 @@ LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci EOF +mkdir -p /etc/compass-ci/account +cat > /etc/compass-ci/account/autosubmit.yaml <<EOF +my_name: Auto Submit Robot +my_email: autosubmit@localhost +my_uuid: $(uuidgen) +EOF + cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci -- 2.23.0

1 0

[PATCH compass-ci] container/assign-account: refactor the way of getting files
by Lu Kaiyi 14 Dec '20

14 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 95bec14..ab00b58 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -64,7 +64,7 @@ class AccountStorage def read_account_info available_dir = File.join(@account_dir, 'available-users') - files = Dir.open(available_dir).to_a + files = Dir.entries(available_dir) files -= ['.', '..'] message = 'no more available users' -- 2.23.0

1 0

[PATCH compass-ci 2/2] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 14 Dec '20

14 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 108 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) diff --git a/doc/tutorial.md b/doc/tutorial.md index 39d62e1..96d4e32 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -146,6 +146,114 @@ job yaml ��[YAML](http://yaml.org/YAM ### �� OS �� +�� initramfs �� cgz �� cgz �� rootfs�� + +#### �� + + 1. �� os �� rootfs�� openEuler �� + - �� docker �� rootfs + + 1) �� openEuler �� docker �� + + ```bash + wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… + ``` + 2) �� docker �� + ```bash + docker load -i openEuler-docker.aarch64 + ``` + + 3) �� openEuler �� + ```bash + docker run -d --name initramfs-docker openeuler-20.03-lts + ``` + 4) �� docker �� rootfs + ```bash + docker cp -a initramfs-docker:/ openEuler-rootfs/ + ``` + + {compass-ci} �� '.tar.xz' �� docker �� rootfs �� + ```bash + ./{compass-ci}/container/docker-rootfs/run openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs-20.03 + ``` + + - �� qemu.img(qcow2��)�� rootfs (openEuler��) + + 1) �� openEuler �� qcow2 �� + ```bash + wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… + ``` + 2) ��{compass-ci}/container/qcow2rootfs ��rootfs + ```bash + cd {compass-ci}/container/qcow2rootfs + ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs + ``` + + 2. ��rootfs + 1. �� chroot �� rootfs (�� root ��) + ```bash + chroot openEuler-rootfs + ``` + 2. �� + + a. �� root �� + b. �� /sbin/init + > ��systemd�� + ```bash + ls -l /sbin/init + /sbin/init -> ../lib/systemd/systemd + ``` + c. �� docker �� osimage �� + > �� docker �� /.dockerenv + d. �� + > openssh-server (ssh ��) + > ntpdate (��) + > cifs-utils (cifs��) + > nfs-utils (nfs��) + > curl (��url��) + ```bash + yum -y install openssh-server ntpdate cifs-utils nfs-utils curl + ``` + e. �� docker �� osimage �� + > �� docker �� /.dockerenv + + 3. ��rootfs + ```bash + cd $rootfs + find . | coip -o -Hnewc |gzip -9 > $os_name.cgz + ``` + + 4. ��/�� + �� {compass-ci} �� rootfs ��/�� + ```bash + find /lib/modules/* | cpio -o -Hnewc | gzip -9 > /modules-$os_name.cgz + find /usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > /headers-$os_name.cgz + ``` + +#### FAQ +1. �� Unable to mount root fs on unknown-block�� + - �� + ```bash + [ 0.390437] List of all partitions: + [ 0.390806] No filesystem could mount root, tried: + [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) + ... + [ 0.399404] Memory Limit: none + [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- + ``` + - �� + + 1�� + 2�� 644 �� + +2. �� + + - �� + + �� + - �� + + �� rootfs �� ### �� -- 2.23.0

2 2

[PATCH lkp-tests] sbin/submit: fix -m when submit job failed
by Wu Zhende 14 Dec '20

14 Dec '20

[Why] When use "submit -m", if submit job failed, the monitor is still run. [Error] submit iperf.yaml failed, got job_id=0, error: Failed to verify the account. Please check your configuration query=>{"job_id":[]} connect to ws://localhost:11310/filter Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/submit | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/submit b/sbin/submit index 6766e704a..05b3e86dd 100755 --- a/sbin/submit +++ b/sbin/submit @@ -162,6 +162,7 @@ ARGV.each do |jobfile| job_ids << msg['job_id'].to_s puts("submit #{jobfile}, got job_id=#{msg['job_id']}") else + opt_monitor = false puts("submit #{jobfile} failed, got job_id=#{msg['job_id']}, error: #{msg['message']}") end end -- 2.23.0

3 2

[PATCH v4 compass-ci 3/3] lib/compare_data_format.rb: limit the number of table columns in a single row
by Lu Kaiyi 14 Dec '20

14 Dec '20

[example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - meminfo.Active - meminfo.Memused filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 4k|1G | 4k|10G | 4k|80G | 4k|100G | 16k|1G | 16k|10G | 16k|100G | 32k|1G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1051645.34 | 704958.00 | 706798.00 | 657669.27 | 1030484.85 | 979289.00 | 948576.00 | 1262591.17 | | standard_deviation openeuler 20.03 | 74.00 | 0.00 | 0.00 | 8.00 | 46.00 | 0.00 | 2.00 | 42.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 64k|1G | 64k|100G | 128k|1G | 128k|100G | 256k|1G | 256k|100G | 512k|1G | 512k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1418731.53 | 1854231.33 | 2004321.76 | 2960171.75 | 2979326.50 | 4775340.00 | 5003555.53 | 5630845.75 | | standard_deviation openeuler 20.03 | 47.00 | 10.00 | 65.00 | 18.00 | 88.00 | 8.00 | 108.00 | 15.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 1024k|1G | 1024k|100G | 2048k|100G | 4096k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 8974190.91 | 9641039.00 | 5480763.67 | 7302509.67 | | standard_deviation openeuler 20.03 | 118.00 | 36.00 | 86.00 | 67.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 42 ++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index c1f7023..aa89d0d 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -72,19 +72,20 @@ end # format compare template results into a table format # class FormatTableData - def initialize(result_hash) + def initialize(result_hash, row_size = 8) @title = result_hash['title'] @tables = result_hash['tables'] @unit = result_hash['unit'] @x_name = result_hash['x_name'] + @row_size = row_size end def show_table @tables.each do |table_title, table| @tb = Terminal::Table.new set_table_title - set_field_names(table_title, table) - add_rows(table) + row_num = get_row_num(table) + split_data_column(table_title, table, row_num) set_align_column print_table end @@ -94,14 +95,37 @@ class FormatTableData @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" end - def set_field_names(table_title, table) + def get_row_num(table) + data_column_size = table['average']['source'][0].size + row_num = data_column_size / @row_size + row_rem = data_column_size % @row_size + if row_rem.positive? + row_num += 1 + end + row_num + end + + def split_data_column(table_title, table, row_num) + row_num.times do |row| + starts = 1 + row * @row_size + ends = starts + @row_size + set_field_names(table_title, table, starts, ends) + add_rows(table, starts, ends) + break if row == row_num - 1 + + @tb.add_separator + @tb.add_separator + end + end + + def set_field_names(table_title, table, starts, ends) field_names = [table_title] - field_names.concat(table['average']['source'][0]) + field_names.concat(table['average']['source'][0][starts - 1...ends - 1]) @tb.add_row(field_names) @tb.add_separator end - def add_rows(table) + def add_rows(table, starts, ends) row_names = %w[average standard_deviation change] max_size = row_names.map(&:size).max row_names.each do |row_name| @@ -109,15 +133,15 @@ class FormatTableData dimensions_size = table[row_name]['dimensions'].size (1...dimensions_size).each do |index| - add_row(table, row_name, index, max_size) + add_row(table, row_name, index, max_size, starts, ends) end end end - def add_row(table, row_name, index, max_size) + def add_row(table, row_name, index, max_size, starts, ends) row = table[row_name]['source'][index] row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') - format_data_row = row[1..-1] + format_data_row = row[starts...ends] if row_name == 'change' format_data_row.map! { |data| format('%.1f%%', data) } else -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] add options for compare_by_template
by Lu Kaiyi 14 Dec '20

14 Dec '20

if want json format results, use command like below: compare -t compare_template.yaml --theme json otherwise, print table format results by default: compare -t compare_template.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 11 ++++++++--- sbin/compare | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 6a60f2b..0e08ec7 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -507,11 +507,16 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) + theme = options[:theme] formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_results = FormatTableData.new(echart_results) - table_results.show_table + if theme == 'json' + print JSON.pretty_generate(echart_results) + else + table_results = FormatTableData.new(echart_results) + table_results.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

1 0