- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 04 Jan '21

04 Jan '21

if user have multi email address, can set mapping to redis queue 'email_mapping' to enable to send mail to a specified address of user with any email address of user in the send address bar example: email_addr_a => email_addr email_addr_b => email_addr when you write either email_addr_a or email_addr_b in the send address bar, the email address will be converted to email_addr Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..60ff3b8 --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# to check if the emails count has beyond the limit +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_mapping', email + mapped_email = @redis.hget 'email_mapping', 'luanshd(a)crystal.ci' + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + end + return mail_list + end +end -- 2.23.0

1 1

[PATCH compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 04 Jan '21

04 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH compass-ci 6/7] lib/mail-post: add email check
by Luan Shengde 04 Jan '21

04 Jan '21

add check email count only email addresses that email count within the limited value can be sent mail add check email mapping if has set email mapping for an email address. the email address will be converted to the mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..eb21ddb --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,31 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + check_email_limit(mail_info.clone) end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +96,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH compass-ci 5/7] send-internet-mail: add email limit queue
by Luan Shengde 04 Jan '21

04 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../email_redis_queue_limit.rb | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100755 container/send-internet-mail/email_redis_queue_limit.rb diff --git a/container/send-internet-mail/email_redis_queue_limit.rb b/container/send-internet-mail/email_redis_queue_limit.rb new file mode 100755 index 0000000..0999983 --- /dev/null +++ b/container/send-internet-mail/email_redis_queue_limit.rb @@ -0,0 +1,48 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# to check if the emails count has beyond the limit +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + end + + if @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + if (@redis.hget 'email_in_limit', email).to_i >= 10 + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 04 Jan '21

04 Jan '21

timing work to reset email queues Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..099d89d --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to enable to send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 04 Jan '21

04 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..2cdfb93 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route |awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_redis_queue_limit' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 04 Jan '21

04 Jan '21

[why] the service need to install redis to store email counts to redis queue the service need to install rufus-scheduler to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 04 Jan '21

04 Jan '21

add x86_64 and riscv64 qemu command and options. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 0c98065..6a1807f 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -25,6 +25,7 @@ qemu_command() { qemu=( qemu-system-aarch64 + qemu-system-x86_64 qemu-kvm ) @@ -74,6 +75,11 @@ check_option_value() } [ -n "$initrds" ] || exit + + [ -n "$(echo $kernel | grep "riscv64")" ] && qemu=qemu-system-riscv64 + + ls $bios >/dev/null 2>&1 || bios=/usr/share/ovmf/OVMF.fd + ls $helper >/dev/null 2>&1 || helper=/usr/lib/qemu/qemu-bridge-helper } get_initrd() @@ -122,6 +128,17 @@ individual_option() -bios $bios -nic $nic ) + + x86_64=( + -bios $bios + -nic $nic + ) + + riscv64=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) } run_qemu() @@ -132,6 +149,13 @@ run_qemu() ;; qemu-kvm) [ "$(arch)" == "aarch64" ] && kvm=("${kvm[@]}" "${aarch64[@]}") + [ "$(arch)" == "x86_64" ] && kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-system-x86_64) + kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-system-riscv64) + kvm=("${kvm[@]}" "${riscv64[@]}") ;; esac -- 2.23.0

2 2

[PATCH v3 compass-ci] container/srv-http: import h5ai
by Lu Weitao 04 Jan '21

04 Jan '21

[Why] Optimize all pages style to make it more modern Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 16 +++++++++++++--- container/srv-http/build | 7 +++++++ .../srv-http/root/etc/nginx/conf.d/default.conf | 5 ++--- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 8f9a36d..987e40e 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,15 +1,25 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM nginx:alpine +FROM alpine:3.9 + ADD root / -RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo +RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm \ + && mkdir /run/nginx/ \ + && mkdir /h5ai RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN chown -R nginx:nginx /srv/_h5ai +WORKDIR /h5ai +COPY h5ai /h5ai + +# build h5ai package +RUN npm install \ + && npm run build \ + && cp -r ./build/_h5ai /srv \ + && chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] diff --git a/container/srv-http/build b/container/srv-http/build index 3a500ea..054e826 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,4 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${H5AI_SRC:=/c/h5ai} + +cp -r $H5AI_SRC ./ + docker build -t srv-http . + +# clean h5ai folder +rm -rf ./h5ai diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 676418a..646eee3 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -3,8 +3,8 @@ server { server_name "srv-http"; server_tokens off; - root /usr/share/nginx/html; - index index.html; + root /srv; + index index.html index.php /_h5ai/public/index.php; charset utf-8,gbk; @@ -24,7 +24,6 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; location ~* /[-_a-z]+$ { add_header Content-Type text/plain; } -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu: refactor kvm.sh
by Liu Yinsi 04 Jan '21

04 Jan '21

before: just support aarch64 after: easy to support more arch(aarch64, x86_64, riscv64). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 51 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..0c98065 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,6 +7,9 @@ : ${nr_cpu:=1} : ${memory:=1G} +bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd +helper=/usr/libexec/qemu-bridge-helper + check_logfile() { log_file=/srv/cci/serial/logs/${hostname} @@ -20,8 +23,15 @@ check_logfile() qemu_command() { - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm + qemu=( + qemu-system-aarch64 + qemu-kvm + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done } write_logfile() @@ -72,6 +82,11 @@ get_initrd() cat $initrds > $initrd } +get_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + print_message() { echo $SCHED_PORT @@ -83,24 +98,42 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) +} + +run_qemu() +{ + case $qemu in + qemu-system-aarch64) + kvm=("${kvm[@]}" "${aarch64[@]}") + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && kvm=("${kvm[@]}" "${aarch64[@]}") + ;; + esac "${kvm[@]}" --append "${append}" } @@ -112,7 +145,11 @@ qemu_command parse_ipxe_script check_option_value get_initrd +get_nic print_message +public_option +individual_option + run_qemu -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 04 Jan '21

04 Jan '21

add x86_64 and riscv64 qemu command and options. test: /srv/result/host-info/2021-01-04/vm-2p8g/debian-sid-aarch64/z9.280125 /srv/result/host-info/2021-01-04/vm-2p8g/debian-sid-x86_64/z9.280114 /srv/result/host-info/2021-01-04/vm-2p8g/debian-sid-riscv64/z9.280135 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 0c98065..1018d06 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -25,6 +25,7 @@ qemu_command() { qemu=( qemu-system-aarch64 + qemu-system-x86_64 qemu-kvm ) @@ -74,6 +75,12 @@ check_option_value() } [ -n "$initrds" ] || exit + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 + + ls $bios >/dev/null 2>&1 || bios=/usr/share/ovmf/OVMF.fd + ls $helper >/dev/null 2>&1 || helper=/usr/lib/qemu/qemu-bridge-helper } get_initrd() @@ -122,6 +129,17 @@ individual_option() -bios $bios -nic $nic ) + + x86_64=( + -bios $bios + -nic $nic + ) + + riscv64=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) } run_qemu() @@ -132,6 +150,13 @@ run_qemu() ;; qemu-kvm) [ "$(arch)" == "aarch64" ] && kvm=("${kvm[@]}" "${aarch64[@]}") + [ "$(arch)" == "x86_64" ] && kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-system-x86_64) + kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-system-riscv64) + kvm=("${kvm[@]}" "${riscv64[@]}") ;; esac -- 2.23.0

3 4

[PATCH v2 lkp-tests] lib/bootstrap.sh: fix error when os is not integrated into lkp-tests
by Yu Chuan 04 Jan '21

04 Jan '21

[Why] If the current os has not been integrated into lkp-tests, run method `detect_system`, then var `$_system_name_lowercase` will be 'unknown'. So we should adapt this. [Error] + local DISTRO=unknown ++ echo ntpdate ++ /home/yuchuan/lkp-tests/sbin/adapt-packages unknown Traceback (most recent call last): 3: from /home/yuchuan/lkp-tests/sbin/adapt-packages:11:in `<main>' 2: from /usr/share/ruby/psych.rb:497:in `load_file' 1: from /usr/share/ruby/psych.rb:497:in `open' /usr/share/ruby/psych.rb:497:in `initialize': No such file or directory @ rb_sysopen - /home/yuchuan/lkp-tests/distro/adaptation/unknown (Errno::ENOENT) [How] If the current os has not been integrated into lkp-tests, then return. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..cbf867ce322f 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -240,6 +240,10 @@ run_ntpdate() has_cmd ntpdate || { detect_system local DISTRO=$_system_name_lowercase + [ "$DISTRO" == "unknown" ] && { + echo "[WARNING] current os cannot be recognized by lkp-test." + return + } . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) } -- 2.23.0

1 1

[PATCH lkp-tests 2/2] hosts/vm-2p32g: add vm-2p32g file
by Lin Jiaxin 04 Jan '21

04 Jan '21

linux compile need big memory Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- hosts/vm-2p32g | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 hosts/vm-2p32g diff --git a/hosts/vm-2p32g b/hosts/vm-2p32g new file mode 100644 index 00000000..14455f20 --- /dev/null +++ b/hosts/vm-2p32g @@ -0,0 +1,5 @@ +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 0

[PATCH lkp-tests 1/2] jobs/build-linux.yaml: optimize the code to make it more reasonable
by Lin Jiaxin 04 Jan '21

04 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/build-linux.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/jobs/build-linux.yaml b/jobs/build-linux.yaml index 1ced05ad..c1d31dca 100644 --- a/jobs/build-linux.yaml +++ b/jobs/build-linux.yaml @@ -2,6 +2,6 @@ suite: build-pkg category: functional build-pkg: config: <%= - today = Time.new - Array.new(100) { |i| 'aarch64-randconfig-' + today.strftime('%Y-%m-%d') + '-' + format('%02d', i) } + today = Time.new.strftime('%Y-%m-%d') + Array.new(100) { |i| "aarch64-randconfig-#{today}-" + format('%02d', i) } %> -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] providers/qemu: refactor kvm.sh
by Liu Yinsi 04 Jan '21

04 Jan '21

before: just support aarch64 after: easy to support more arch(aarch64, x86_64, riscv64). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 51 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..0c98065 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,6 +7,9 @@ : ${nr_cpu:=1} : ${memory:=1G} +bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd +helper=/usr/libexec/qemu-bridge-helper + check_logfile() { log_file=/srv/cci/serial/logs/${hostname} @@ -20,8 +23,15 @@ check_logfile() qemu_command() { - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm + qemu=( + qemu-system-aarch64 + qemu-kvm + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done } write_logfile() @@ -72,6 +82,11 @@ get_initrd() cat $initrds > $initrd } +get_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + print_message() { echo $SCHED_PORT @@ -83,24 +98,42 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) +} + +run_qemu() +{ + case $qemu in + qemu-system-aarch64) + kvm=("${kvm[@]}" "${aarch64[@]}") + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && kvm=("${kvm[@]}" "${aarch64[@]}") + ;; + esac "${kvm[@]}" --append "${append}" } @@ -112,7 +145,11 @@ qemu_command parse_ipxe_script check_option_value get_initrd +get_nic print_message +public_option +individual_option + run_qemu -- 2.23.0

1 0

[PATCH lkp-tests] jobs/build-linux.yaml: submit 100 randconfig job
by Lin Jiaxin 04 Jan '21

04 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/build-linux.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/jobs/build-linux.yaml b/jobs/build-linux.yaml index 080a84b3..1ced05ad 100644 --- a/jobs/build-linux.yaml +++ b/jobs/build-linux.yaml @@ -1,4 +1,7 @@ suite: build-pkg category: functional build-pkg: -config: <%= 'aarch64-randconfig-'+SecureRandom.random_number(1000).to_s %> +config: <%= + today = Time.new + Array.new(100) { |i| 'aarch64-randconfig-' + today.strftime('%Y-%m-%d') + '-' + format('%02d', i) } + %> -- 2.23.0

2 1

[PATCH lkp-tests] lib/bootstrap.sh: fix error when os is not integrated into lkp-tests
by Yu Chuan 04 Jan '21

04 Jan '21

[Why] If the current os has not been integrated into lkp-tests, run method `detect_system`, then var `$_system_name_lowercase` will be 'unknown'. So we should adapt this. [Error] + local DISTRO=unknown ++ echo ntpdate ++ /home/yuchuan/lkp-tests/sbin/adapt-packages unknown Traceback (most recent call last): 3: from /home/yuchuan/lkp-tests/sbin/adapt-packages:11:in `<main>' 2: from /usr/share/ruby/psych.rb:497:in `load_file' 1: from /usr/share/ruby/psych.rb:497:in `open' /usr/share/ruby/psych.rb:497:in `initialize': No such file or directory @ rb_sysopen - /home/yuchuan/lkp-tests/distro/adaptation/unknown (Errno::ENOENT) [How] If the current os has not been integrated into lkp-tests, then return. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..ab6573af1a7a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -240,6 +240,7 @@ run_ntpdate() has_cmd ntpdate || { detect_system local DISTRO=$_system_name_lowercase + [ "$DISTRO" == "unknown" ] && return . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) } -- 2.23.0

2 4

[PATCH v2 compass-ci] container/srv-http: import h5ai
by Lu Weitao 04 Jan '21

04 Jan '21

[Why] Optimize all pages style to make it more modern Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 16 +++++++++++++--- container/srv-http/build | 7 +++++++ .../srv-http/root/etc/nginx/conf.d/default.conf | 5 ++--- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 8f9a36d..987e40e 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,15 +1,25 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM nginx:alpine +FROM alpine:3.9 + ADD root / -RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo +RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm \ + && mkdir /run/nginx/ \ + && mkdir /h5ai RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN chown -R nginx:nginx /srv/_h5ai +WORKDIR /h5ai +COPY h5ai /h5ai + +# build h5ai package +RUN npm install \ + && npm run build \ + && cp -r ./build/_h5ai /srv \ + && chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] diff --git a/container/srv-http/build b/container/srv-http/build index 3a500ea..7b0ea74 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,4 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $H5AI_SRC ]] || H5AI_SRC=/c/h5ai + +cp -r $H5AI_SRC ./ + docker build -t srv-http . + +# clean h5ai folder +rm -rf ./h5ai diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 676418a..646eee3 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -3,8 +3,8 @@ server { server_name "srv-http"; server_tokens off; - root /usr/share/nginx/html; - index index.html; + root /srv; + index index.html index.php /_h5ai/public/index.php; charset utf-8,gbk; @@ -24,7 +24,6 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; location ~* /[-_a-z]+$ { add_header Content-Type text/plain; } -- 2.23.0

2 2

[PATCH v4 compass-ci 2/2] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 04 Jan '21

04 Jan '21

[why] make es result more easy access. [example] input: result_hash = {} es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 29d50e9..63d32ce 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -129,7 +129,7 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + parse_fields(result) end end @@ -217,3 +217,47 @@ def build_aggs_from_fields(fields, aggs_hash) build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) end + +# input: +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result) + result_hash ||= {} + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + all_field = result[sub_field]['buckets'] + result_hash[key] = parse_fields(all_field) + else + result_hash[key] = result['doc_count'] + end + end + + result_hash +end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 04 Jan '21

04 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 71 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..29d50e9 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,50 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = {} + build_aggs_from_fields(fields, aggs_hash) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +190,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# aggs_hash = {} +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields, aggs_hash) + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] = { + "all_#{field}" => { + 'terms' => { field: field, size: 1000 } + } + } + + build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) +end -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootfs/common: add comment for set password file.
by Wang Chenglong 04 Jan '21

04 Jan '21

[Why] add password file is easy to ignore. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common index fe37da9..2ee16c8 100755 --- a/container/docker-rootfs/common +++ b/container/docker-rootfs/common @@ -33,7 +33,8 @@ usage() dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: - ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists " } -- 2.23.0

3 4

[PATCH v3 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 04 Jan '21

04 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..1b6f42f --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +## Use the tool to create a new nfsroot for os + +## Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +## Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists + +## Some configuration items: + ./packages-to-install + # If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + # Set the password for the image into this file. -- 2.23.0

2 2

[PATCH v4 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 04 Jan '21

04 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..e7b5084 --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +# Use the tool to create a new nfsroot for os + +## Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +## Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # Please ensure $HOME/.config/compass-ci/rootfs-passwd exists. + +## Some configuration items: + ./packages-to-install + # If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + # Set the password for the image into this file. -- 2.23.0

1 0

[PATCH compass-ci 2/2] qcow2rootfs: fix rootfs-dir permission error
by Yu Chuan 03 Jan '21

03 Jan '21

[Error Msg] '01/03/21_04:25:49' Compressing file to /root/extract/rootfs-dir/qcow2-out.tar.gz Please wait until command returned... <r.gz compress:gzip numericowner:true xattrs:true selinux:true acls:true libguestfs: error: /root/extract/rootfs-dir/qcow2-out.tar.gz: Permission denied libguestfs: error: file receive cancelled by daemon Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/qcow2rootfs/bin/main | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/main b/container/qcow2rootfs/bin/main index b6ef7238f24a..c7527b794b89 100755 --- a/container/qcow2rootfs/bin/main +++ b/container/qcow2rootfs/bin/main @@ -34,7 +34,7 @@ unzip_qcow2_xz() { extract_rootfs() { echo "[WARNNING] Temporarily change owner of qcow2 file" chown "$EXTRACT_USER":"$EXTRACT_USER" "$QCOW2_FILE" - chmod 775 "$ROOTFS_DIR" + chmod 777 "$ROOTFS_DIR" echo "[$(date +'%D %T')] Converting .qcow2 to .tar.gz, please wait several minutes..." su - "$EXTRACT_USER" -c " -- 2.23.0

1 0

[PATCH compass-ci 1/2] qcow2rootfs: generate pkgs record file for result rootfs
by Yu Chuan 03 Jan '21

03 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/qcow2rootfs/bin/cci-generate-pkgs | 15 +++++++++++++++ container/qcow2rootfs/bin/main | 6 ++++++ 2 files changed, 21 insertions(+) create mode 100755 container/qcow2rootfs/bin/cci-generate-pkgs diff --git a/container/qcow2rootfs/bin/cci-generate-pkgs b/container/qcow2rootfs/bin/cci-generate-pkgs new file mode 100755 index 000000000000..e6a974cd0f41 --- /dev/null +++ b/container/qcow2rootfs/bin/cci-generate-pkgs @@ -0,0 +1,15 @@ +#!/bin/bash + +RESULT_FILE=/etc/pkgs-$(date "+%Y-%m-%d-%H-%M-%S").list + +command -v rpm >/dev/null && { + rpm -qa > $RESULT_FILE +} + +command -v dpkg >/dev/null && { + dpkg -l > $RESULT_FILE +} + +chmod 644 $RESULT_FILE + +exit 0 diff --git a/container/qcow2rootfs/bin/main b/container/qcow2rootfs/bin/main index b70ec7766649..b6ef7238f24a 100755 --- a/container/qcow2rootfs/bin/main +++ b/container/qcow2rootfs/bin/main @@ -74,6 +74,12 @@ pre_config_rootfs() { if [ -f ${ROOTFS_DIR}/etc/selinux/config ]; then sed -i -r 's/SELINUX=enforcing/SELINUX=disabled/g' "$ROOTFS_DIR/etc/selinux/config" fi + + # generate /etc/pkgs-$(date "+%Y-%m-%d-%H-%M-%S").list file for result rootfs + echo "generate /etc/pkgs-$(date "+%Y-%m-%d-%H-%M-%S").list" + cp $EXTRACT_ROOT/bin/cci-generate-pkgs $ROOTFS_DIR + chroot $ROOTFS_DIR /cci-generate-pkgs + rm /$ROOTFS_DIR/cci-generate-pkgs } clean_environment() { -- 2.23.0

1 0