December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/2] sparrow: refactor the code for docker build all image concurrently
by Yu Chuan 16 Dec '20

16 Dec '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/4-docker/buildall | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index eb245008ad45..a9269e03a550 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -8,18 +8,14 @@ export action=$1 build_depends() { local container=$1 - local should_wait=$2 for dep in $(cat $container/*-depends 2> /dev/null) do - build_depends $CONTAINER_PATH/$dep block_wait + build_depends $CONTAINER_PATH/$dep & done + wait - if [ -n "$should_wait" ]; then - do_one $container - else - do_one $container & - fi + do_one $container } do_one() @@ -44,8 +40,8 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir + build_depends $dir & done - wait + rm -fr $tmpdir -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] sparrow: fix that 0-package/os/openEuler can only be executed once
by Yu Chuan 16 Dec '20

16 Dec '20

[Error Msg] ++ rpm -i lib64lockfile1-1.16-1-omv4002.aarch64.rpm lockfile-progs-0.1.18-5-omv4002.aarch64.rpm warning: lib64lockfile1-1.16-1-omv4002.aarch64.rpm: Header V4 RSA/SHA256 Signature, key ID bf81de15: NOKEY package lib64lockfile1-1.16-1.aarch64 is already installed file /usr/lib64/liblockfile.so.1.0 from install of lib64lockfile1-1.16-1.aarch64 conflicts with file from package lib64lockfile1-1.16-1.aarch64 package lockfile-progs-0.1.18-5.aarch64 is already installed file /usr/bin/lockfile-check from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-create from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-remove from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-touch from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-lock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-touchlock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-unlock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 [How] yuchuan@localhost ~% rpm --help | grep replacepkgs --force short hand for --replacepkgs --replacefiles --replacepkgs reinstall if the package is already present Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/os/openEuler | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 43e96ab8ab2e..813b0ad3c40a 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -27,5 +27,5 @@ yum install -y --skip-broken "${pkgs[@]}" wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… -rpm -i lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm +rpm -i --replacepkgs lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm rm -rf *.rpm -- 2.23.0

2 1

[PATCH compass-ci] sparrow/5-build: adapted for x86_64 machine
by Liu Yinsi 16 Dec '20

16 Dec '20

before: can build ipxe files on aarch64 machine after: can build ipxe files on aarch64 and x86_64 machine Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/ipxe | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/sparrow/5-build/ipxe b/sparrow/5-build/ipxe index a66e298..0f99fa0 100755 --- a/sparrow/5-build/ipxe +++ b/sparrow/5-build/ipxe @@ -15,11 +15,25 @@ cd ipxe/src || exit cpus=$(nproc) -make ARCH=arm64 bin-arm64-efi/ipxe.efi -make ARCH=arm64 bin-arm64-efi/snponly.efi -make ARCH=arm64 bin-arm64-efi/snp.efi -make CONFIG=rpi bin-arm64-efi/rpi.efi - -mkdir -p /tftpboot/ipxe/bin-arm64-efi/ -cp -a bin-arm64-efi/*.efi /tftpboot/ipxe/bin-arm64-efi/ -chmod 664 /tftpboot/ipxe/bin-arm64-efi/*.efi +build_ipxe() +{ + make ARCH=$1 bin-$1-efi/ipxe.efi + make ARCH=$1 bin-$1-efi/snponly.efi + make ARCH=$1 bin-$1-efi/snp.efi + make CONFIG=rpi bin-$1-efi/rpi.efi + + mkdir -p /tftpboot/ipxe/bin-$1-efi/ + cp -a bin-$1-efi/*.efi /tftpboot/ipxe/bin-$1-efi/ + chmod 664 /tftpboot/ipxe/bin-$1-efi/*.efi +} + +ARCH=$(arch) + +case "$ARCH" in + "aarch64") + build_ipxe arm64 + ;; + "x86_64") + build_ipxe x86_64 + ;; +esac -- 2.23.0

2 2

[PATCH v3 compass-ci 5/6] fetch-mail: build file
by Luan Shengde 16 Dec '20

16 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/fetch-mail/build diff --git a/container/fetch-mail/build b/container/fetch-mail/build new file mode 100755 index 0000000..e72bed8 --- /dev/null +++ b/container/fetch-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t fetch-mail:latest . -- 2.23.0

1 0

[PATCH v3 compass-ci 2/6] fetch-mail: config file for procmail
by Luan Shengde 16 Dec '20

16 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/procmailrc.template | 25 ++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 container/fetch-mail/procmailrc.template diff --git a/container/fetch-mail/procmailrc.template b/container/fetch-mail/procmailrc.template new file mode 100644 index 0000000..9cea4bc --- /dev/null +++ b/container/fetch-mail/procmailrc.template @@ -0,0 +1,25 @@ +DROPPRIVS=yes +PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin +MAILDIR=FETCH_MAIL_DIR +MAILLIST=${MAILDIR} +DEFAULT=${MAILDIR}/FETCH_MAIL_BOX/ +ORGMAIL=${DEFAULT}/FETCH_MAIL_BOX/ +LOGFILE=${MAILDIR}/log/procmail # recommended + +UMASK=002 + +DISPLAY=:0 +XAUTHORITY=$HOME/.Xauthority + +# One often receives duplicate messages for cross-postings to mailing +# lists, or explicit replies to a post from you in addition to the +# mailing list message. +# +# The following recipe eliminates all duplicate messages. It tells +# formail(1) to keep a cache file of the Message-IDs of the most recent +# messages you've received. Since Message-IDs are guaranteed unique for +# distinct messages, they are ideally suited to weed out duplicates. + +# LOCKFILE=$MAILDIR/.msgid.lock +:0 Wh: $MAILDIR/.msgid.lock +| formail -D 655360 $MAILDIR/log/msgid -- 2.23.0

1 0

[PATCH v3 compass-ci 1/6] fetch-mail: config file for fetchmail
by Luan Shengde 16 Dec '20

16 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/fetchmailrc.template | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/fetch-mail/fetchmailrc.template diff --git a/container/fetch-mail/fetchmailrc.template b/container/fetch-mail/fetchmailrc.template new file mode 100644 index 0000000..8c7e82b --- /dev/null +++ b/container/fetch-mail/fetchmailrc.template @@ -0,0 +1,5 @@ +poll imap.qq.com with proto IMAP +username "FETCH_MAIL_ADDRESS" +password "FETCH_MAIL_AUTH_CODE" +mda "/usr/bin/procmail -f %F -d %T" +nokeep -- 2.23.0

1 0

[PATCH v5 compass-ci 1/3] lib: friendly show compare template result in command line
by Lu Kaiyi 16 Dec '20

16 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the results of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | average openeuler 20.03 | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | standard_deviation openeuler 20.03 | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | average centos 7.6 | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | average openeuler 20.03 | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | standard_deviation centos 7.6 | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | | standard_deviation openeuler 20.03 | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | | change centos 7.6 vs openeuler 20.03 | 183.0% | 325.6% | 97.4% | 150.5% | 219.2% | 138.3% | 278.0% | 211.9% | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 73 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 +- 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..c1f7023 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' + # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +68,77 @@ class FormatEchartData end end +# ---------------------------------------------------------------------------------------------------- +# format compare template results into a table format +# +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + @unit = result_hash['unit'] + @x_name = result_hash['x_name'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_table_title + set_field_names(table_title, table) + add_rows(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_rows(table) + row_names = %w[average standard_deviation change] + max_size = row_names.map(&:size).max + row_names.each do |row_name| + next unless table[row_name] + + dimensions_size = table[row_name]['dimensions'].size + (1...dimensions_size).each do |index| + add_row(table, row_name, index, max_size) + end + end + end + + def add_row(table, row_name, index, max_size) + row = table[row_name]['source'][index] + row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') + format_data_row = row[1..-1] + if row_name == 'change' + format_data_row.map! { |data| format('%.1f%%', data) } + else + format_data_row.map! { |data| format('%.2f', data) } + end + @tb.add_row([row_title, *format_data_row]) + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5069939..9bf676f 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -511,7 +511,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_results = FormatTableData.new(echart_results) + table_results.show_table end # Format Fields -- 2.23.0

1 0

[PATCH v3 compass-ci] docker-rootfs/setup-image: set the system time zone
by Wang Chenglong 16 Dec '20

16 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 8e2e88a..787a42c 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -13,6 +13,11 @@ remove_file() rm -rf /.dockerenv } +set_localtime() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + setup_login() { [ -n "$ROOT_PASSWD" ] || return @@ -32,5 +37,6 @@ pack_cgz() install_packages remove_file +set_localtime setup_login pack_cgz -- 2.23.0

1 0

[PATCH v5 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 16 Dec '20

16 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 87 ++++++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 41 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..3110685 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -152,29 +152,20 @@ job yaml ��[YAML](http://yaml.org/YAM 1. �� os �� rootfs�� openEuler �� - �� docker �� rootfs - + 1) �� openEuler �� docker �� - + ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` + ``` + 2) {compass-ci}/container/docker-rootfs �� '.tar.xz' �� docker �� rootfs + ```bash + ./{compass-ci}/container/docker-rootfs/run openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs-20.03 + ``` - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� + + 1) �� openEuler �� qcow2 �� ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… ``` @@ -183,45 +174,59 @@ job yaml ��[YAML](http://yaml.org/YAM cd {compass-ci}/container/qcow2rootfs ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) + + 2. ��rootfs + 1. �� chroot �� rootfs (�� root ��) ```bash chroot openEuler-rootfs ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 2. �� + a. �� /sbin/init + > ��systemd�� + ```bash + ls -l /sbin/init + /sbin/init -> ../lib/systemd/systemd + ``` + b. �� /etc/localtime + > �� + ```bash + ls -l /etc/localtime + /etc/localtime -> ../usr/share/zoneinfo/Asia/Shanghai + ``` + c. �� docker �� osimage �� + > docker �� /.dockerenv �� + + 3. ��rootfs + ```bash + cd $rootfs + find . | coip -o -Hnewc |gzip -9 > $os_name.cgz + ``` + + 4. ��/�� + �� {compass-ci} �� rootfs ��/�� + ```bash + find /lib/modules/* | cpio -o -Hnewc | gzip -9 > /modules-$os_name.cgz + find /usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > /headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH compass-ci] sparrow: fix that 0-package/os/openEuler can only be executed once
by Yu Chuan 16 Dec '20

16 Dec '20

[Error Msg] ++ rpm -i --replacepkgs lib64lockfile1-1.16-1-omv4002.aarch64.rpm lockfile-progs-0.1.18-5-omv4002.aarch64.rpm warning: lib64lockfile1-1.16-1-omv4002.aarch64.rpm: Header V4 RSA/SHA256 Signature, key ID bf81de15: NOKEY file /usr/lib64/liblockfile.so.1.0 from install of lib64lockfile1-1.16-1.aarch64 conflicts with file from package lib64lockfile1-1.16-1.aarch64 file /usr/bin/lockfile-check from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-create from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-remove from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/lockfile-touch from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-lock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-touchlock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 file /usr/bin/mail-unlock from install of lockfile-progs-0.1.18-5.aarch64 conflicts with file from package lockfile-progs-0.1.18-5.aarch64 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/os/openEuler | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 813b0ad3c40a..13e087599218 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -27,5 +27,5 @@ yum install -y --skip-broken "${pkgs[@]}" wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… -rpm -i --replacepkgs lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm +rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm rm -rf *.rpm -- 2.23.0

1 0