- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 07 Jan '21

07 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..9891ae9 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit +# email_in_limit is used to store email address email count +# less than the limit value +# email_out_limit is used to store email address email count +# beyond the limit +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1 +# case the email in email_in_limit but email count less than the +# limit value, its value will +1 +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

3 3

[PATCH lkp-tests] jobs/ebizzy.yaml: modify the value of iterations
by Hu Xuejiao 07 Jan '21

07 Jan '21

[why] seq: invalid floating point argument: 100 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/ebizzy.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/ebizzy.yaml b/jobs/ebizzy.yaml index 9a3e9a16..cc9994c8 100644 --- a/jobs/ebizzy.yaml +++ b/jobs/ebizzy.yaml @@ -3,7 +3,7 @@ testcase: ebizzy category: benchmark nr_threads: 200% -iterations: 100x +iterations: 100 ebizzy: duration: 10s -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/2] stats/crystal/kernbench.cr: create kernbench.cr as same content with stats/kernbench
by Lu Kaiyi 07 Jan '21

07 Jan '21

create kernbench.cr in crystal directory from stats/kernbench first, and then, the second patch will know what has changed from original file. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/kernbench.cr | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 stats/crystal/kernbench.cr diff --git a/stats/crystal/kernbench.cr b/stats/crystal/kernbench.cr new file mode 100644 index 000000000..0da37fe88 --- /dev/null +++ b/stats/crystal/kernbench.cr @@ -0,0 +1,28 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +$stdin.each_line do |line| + case line + when /Elapsed Time/ + puts 'elapsed_time: ' + line.split[2] + puts 'elapsed_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /User Time/ + puts 'user_time: ' + line.split[2] + puts 'user_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /System Time/ + puts 'system_time: ' + line.split[2] + puts 'system_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Percent CPU/ + puts 'percent_cpu: ' + line.split[2] + puts 'percent_cpu_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Context Switches/ + puts 'context_switches: ' + line.split[2] + puts 'context_switches_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Sleeps/ + puts 'sleeps: ' + line.split[1] + puts 'sleeps_stddev: ' + line.split[2].gsub(/[()]/, "") + end +end -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add openeuler_docker.yaml
by Bai Jing 07 Jan '21

07 Jan '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/openeuler_docker.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/openeuler_docker.yaml diff --git a/jobs/openeuler_docker.yaml b/jobs/openeuler_docker.yaml new file mode 100644 index 000000000..376ece156 --- /dev/null +++ b/jobs/openeuler_docker.yaml @@ -0,0 +1,5 @@ +suite: openeuler_docker +category: benchmark +openeuler_docker: + dockerfile_url: + - CentOS/CentOS-Dockerfiles.git -- 2.23.0

2 4

[PATCH v2 lkp-tests] jobs: add openeuler_docker.yaml
by Bai Jing 07 Jan '21

07 Jan '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/openeuler_docker.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/openeuler_docker.yaml diff --git a/jobs/openeuler_docker.yaml b/jobs/openeuler_docker.yaml new file mode 100644 index 000000000..a5db34eef --- /dev/null +++ b/jobs/openeuler_docker.yaml @@ -0,0 +1,6 @@ +suite: openeuler_docker +category: benchmark + +openeuler_docker: + dockerfile_url: + - CentOS/CentOS-Dockerfiles.git -- 2.23.0

1 0

[PATCH lkp-tests] test/email: get username from my_email
by Wei Jihui 07 Jan '21

07 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/email | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/email b/tests/email index 40b48fd9..a5f87d42 100755 --- a/tests/email +++ b/tests/email @@ -15,7 +15,7 @@ data_failure() data="To: $my_email Subject: [NOTIFY Compass-ci] Test job failed -Dear $my_name: +Dear $my_username: Sorry to inform you that your test job is failed. you can click the following link to view details. http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root @@ -33,6 +33,7 @@ compose_email() run_email() { [ -n "$my_email" ] || return 0 + my_username=$(echo "$my_email" | awk -F '@' '{print $1}') compose_email curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" } -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] tests/host-info: computing vm memory
by Wei Jihui 07 Jan '21

07 Jan '21

[why] it can not compute vm memory when vm memory less than 16G. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/host-info | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/host-info b/tests/host-info index fe55d1ff..2721648c 100755 --- a/tests/host-info +++ b/tests/host-info @@ -4,8 +4,8 @@ get_memory() { memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024/16}') - memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G echo "memory: $memory" } -- 2.23.0

1 0

[PATCH compass-ci] 4-docker/buildall: modify the logic
by Hu Xuejiao 07 Jan '21

07 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index ca4406b..ee257b0 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd $container [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] && [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

2 1

[PATCH v4 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 07 Jan '21

07 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 [how] put public options like $kernel, $initrd, $smp together, list different options like $machine, $bios, $nic according to different qemu command, then merge all options. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 106 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 93 insertions(+), 13 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..8749568 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -72,6 +66,50 @@ get_initrd() cat $initrds > $initrd } +get_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +get_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +get_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +get_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +get_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +get_qemu() +{ + qemu=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +121,78 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) + ;; + esac + + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value + get_initrd +get_bios +get_helper +get_nic +get_device +get_netdev +get_qemu print_message +public_option run_qemu -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow: add setup.yaml
by Liu Yinsi 07 Jan '21

07 Jan '21

[why] before deploy compass-ci, fill configuration setup.yaml firstly, will make the whole process cleaner. fill my_name and my_email, it's necessary to verify account when submit job. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/setup.yaml | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 sparrow/setup.yaml diff --git a/sparrow/setup.yaml b/sparrow/setup.yaml new file mode 100644 index 0000000..b1cac8c --- /dev/null +++ b/sparrow/setup.yaml @@ -0,0 +1,11 @@ +# please copy this template with the below commands, +# then fill the configuration according to comments. +# mkdir -p /etc/compass-ci/ +# cp setup.yaml /etc/compass-ci/setup.yaml +# vim /etc/compass-ci/setup.yaml + +# account name +my_name: xxxxxx + +# email address +my_email: xxx@xxx -- 2.23.0

1 0