- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests 1/2] jobs: add test-tools.yaml
by Li Ping 20 Oct '20

20 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- jobs/test-tools.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 jobs/test-tools.yaml diff --git a/jobs/test-tools.yaml b/jobs/test-tools.yaml new file mode 100644 index 00000000..153fefd0 --- /dev/null +++ b/jobs/test-tools.yaml @@ -0,0 +1,10 @@ +suite: test-tools +testcase: test-tools +category: functional + +os_mount: cifs +os_version: 20.03-dailybuild + +test-tools: + testsuite: + case: -- 2.23.0

2 1

[PATCH lkp-tests] bin: mkdir /etc/pki/ca-trust/source/anchors
by Hu Xuejiao 20 Oct '20

20 Oct '20

When run job-task, appeat warning: cannot create /etc/pki/ca-trust/source/anchors/docker_registry_proxy.crt: Directory nonexistent Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index d1b17a9c..09930990 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -139,6 +139,7 @@ install_docker_proxy() Environment="HTTPS_PROXY=http://$DOCKER_PROXY_HOST:$DOCKER_PROXY_PORT/" EOF + mkdir -p /etc/pki/ca-trust/source/anchors curl http://$DOCKER_PROXY_HOST:$DOCKER_PROXY_PORT/ca.crt > \ /etc/pki/ca-trust/source/anchors/docker_registry_proxy.crt || return update-ca-trust extract -- 2.23.0

1 0

[PATCH lkp-tests] bin: update ~ to /root
by Hu Xuejiao 20 Oct '20

20 Oct '20

When run job-task, appear warning: /lkp/lkp/src/bin/lkp-setup-rootfs: 154: cannot create ~/.gemrc: Directory nonexistent Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- bin/lkp-setup-rootfs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 09f7e7eb..d1b17a9c 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -151,7 +151,7 @@ EOF modify_gem_source() { - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + echo ':sources: ["http://rubygems.org"]' >> /root/.gemrc return 0 } -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

cluster nodes need communicate with each other, so set 2 fields: 1) direct_macs: get from cluster file ($LKP_SRC/cluster/*) 2) direct_ips: set by scheduler default (they're local ips) It will add ips to corresponding devices before nodes executing jobs. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..62b7b9d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,11 @@ class Sched # collect all job ids job_ids = [] of String + # set net id, and host id + # ip = net_id + set_id + net_id = "192.168.211." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +224,16 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + direct_macs.size.times do + direct_ips << "#{net_id}#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 1

[PATCH lkp-tests 2/2] delete the field: testcase
by Li Ping 20 Oct '20

20 Oct '20

[why] there is no need to add the field: testcase in the $job.yaml Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 2 +- lib/bootstrap.sh | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 09f7e7eb..05a3325c 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -51,7 +51,7 @@ job_does_not_need_reboot() } grep -q "^do_not_reboot_for_same_kernel: 1$" $job && - is_same_kernel_and_rootfs && is_same_testcase && is_same_bp_memmap && { + is_same_kernel_and_rootfs && is_same_suite && is_same_bp_memmap && { echo "LKP: [do_not_reboot_for_same_kernel] start a new job without rebooting" return 0 } diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 2506c021..a7776d68 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -631,12 +631,12 @@ is_same_kernel_and_rootfs() return 1 } -is_same_testcase() +is_same_suite() { - local current_testcase=$testcase - local next_testcase=$(awk '/^testcase: /{print $2}' $job | tr -d '"') + local current_suite=$suite + local next_suite=$(awk '/^suite: /{print $2}' $job | tr -d '"') - [ "$current_testcase" = "$next_testcase" ] + [ "$current_suite" = "$next_suite" ] } is_same_bp_memmap() -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/host-info b/tests/host-info index a57855dc..9ca06235 100755 --- a/tests/host-info +++ b/tests/host-info @@ -80,6 +80,12 @@ get_partitions() fi } +get_ipmi_ip() +{ + imp_ip=$(ipmitool lan print |awk 'NR==3{print $4}') + [ $? -eq 0 ] && echo "ipmi_ip:" $imp_ip +} + main() { get_node @@ -87,6 +93,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

4 3

[PATCH lkp-tests] sshd: send emails to login services for users
by Zhang Yale 20 Oct '20

20 Oct '20

[Why] Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 99 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 74 insertions(+), 25 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..cd57be4f 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key -# email +# my_email +# my_uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 +[ -n "$ssh_port_len" ] || ssh_port_len=2000 +[ -n "$ssh_port_base" ] || ssh_port_base=50000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") - data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh - -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: - - Login: - ssh root@$PUB_IP - Due time: - $deadline - - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch + data="To: $my_email +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-ci +" +} + +data_failure() +{ + data="To: $my_email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $my_username: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +set_port() +{ + [ -n "$my_uuid" ] || return + + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$ssh_port_len"+"$ssh_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] && return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$my_email" ] && [ -n "$my_uuid" ] || return 0 + my_username=$(echo "$my_email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +set_port +report_ssh_port $port run_email -- 2.23.0

3 5

[PATCH v9 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 19 Oct '20

19 Oct '20

Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 89 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..55015b40 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key # email +# my_uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=30000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $my_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch - Login: - ssh root@$PUB_IP - Due time: - $deadline +Regards +Compass-ci +" +} + +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +Dear $my_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +set_port() +{ + [ -n "$my_uuid" ] || return + + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] && return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$my_uuid" ] || return 0 + my_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +set_port +report_ssh_port $port run_email -- 2.23.0

3 3

[PATCH compass-ci] job.cr: initramfs and nfs use the same kernel file
by Xu Xijian 19 Oct '20

19 Oct '20

Actually the vmlinuz file in SRV_INITRD is copy from SRV_OS, so they are the same file, just use the same path is enough. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..fc15441 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -272,18 +272,9 @@ class Job return true end - private def vmlinuz - if os_mount == "initramfs" - return "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/vmlinuz")}" - else - return "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" - end - end - private def set_kernel_uri - self["kernel_uri"] = "kernel #{vmlinuz()}" + self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" end private def kernel_common_params -- 2.23.0

1 0

[PATCH lab-z9] allot/idle: remove the field -hi1620 from tbox_group
by Li Ping 19 Oct '20

19 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml index 4525fcd..513b631 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml @@ -8,11 +8,11 @@ os_mount: initramfs simplify-ci: multi-qemu-0: nr_vm: 20 - tbox_group: vm-hi1620-2p8g + tbox_group: vm-2p8g multi-qemu-1: nr_vm: 2 - tbox_group: vm-hi1620-2p4g + tbox_group: vm-2p4g multi-qemu-2: nr_vm: 2 - tbox_group: vm-hi1620-1p1g + tbox_group: vm-1p1g sleep: 100d -- 2.23.0

1 0

[PATCH v7 compass-ci 1/6] container/mail-robot: mail-robot.rb
by Luan Shengde 19 Oct '20

19 Oct '20

mail robot for email application monitor mailbox check new added email file in new check new mail's subject return if unmatched subject call answerback-email check to apply uuid/account Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/mail-robot.rb | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..70b8b25 --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'answerback-email.rb' + +mail_inbox = '/srv/cci/Maildir/.compass-ci/new/' +mail_drafts = '/srv/cci/Maildir/.compass-ci/cur/' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + unless added.empty? + added.each do |mail_file| + begin + check_to_send_account(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + end + listener.start + sleep +end + +def check_to_send_account(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + subject = mail_content.subject + return unless subject =~ /apply account/i + + assign_uuid = AssignUuid.new(mail_content) + assign_uuid.send_account + FileUtils.mv(mail_file, mail_drafts) +end + +monitor_dir(mail_inbox, mail_drafts) -- 2.23.0

3 4

[PATCH lkp-tests] lib/job.rb: load hosts file for atomic job
by Wei Jihui 19 Oct '20

19 Oct '20

[why] for commit c725e0f022b7: def each_jobs(&block) each_job_init - load_hosts_config job = deepcopy(@job) @job2 = {} load_defaults + load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| job = deepcopy(@job) this @job has no load hosts config, so it should do it under "@jobs.each do |hash|" for atomic job. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..6359254e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -696,9 +696,11 @@ class Job each_job_init each_job(&block) @jobs.each do |hash| + @load_hosts_done = false @job = deepcopy(job) @job2 = hash load_defaults + load_hosts_config each_job_init each_job(&block) end -- 2.23.0

2 2

[PATCH v7 compass-ci 5/6] container/mail-robot: start file
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/start | 39 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..108ed83 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_INTERNET_MAIL_PORT +] + +defaults = relevant_defaults(names) + +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] +SEND_MAIL_PORT = defaults['SEND_INTERNET_MAIL_PORT'] + +cmd = %W[ + docker run + --restart=always + --name=mail-robot + -d + -e J_HOST=#{JUMPER_HOST} + -e J_PORT=#{JUMPER_PORT} + -e S_M_PORT=#{SEND_MAIL_PORT} + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir:/srv/cci/Maildir:rw + -v /c/compass-ci:/c/compass-ci:ro + mail-robot +] + +cmd += ['bash', '-c', 'umask 002 && ruby ./mail-robot.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v2 compass-ci] sched: throw an exception when deps/pkg does not exist
by Cao Xueliang 19 Oct '20

19 Oct '20

When submitting a job, an exception is thrown when deps/pkg does not exist on the server. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..7785c2a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -389,10 +389,12 @@ class Job program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") && File.exists?(pkg_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

2 5

[PATCH v2 lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 19 Oct '20

19 Oct '20

if get string named with *.yaml from ARGV, treat it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..ce8accf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /^.+\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/2] feat(iso2rootfs): modify the rootfs dir layout and rootfs dir name
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] 1. Modify the dir structure of dailybuild rootfs dir. 2. Modify the dir name of dailybuild rootfs dir. [How] before: "{os}/{os_arch}/debug-versions/dailybuild/{iso_version}-%Y%m%d%H%M%S" after: "{os}/{os_arch}/{iso_version}-%Y-%m-%d-%H-%M-%S" Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 524ab6695f93..89328f2f4a9b 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,6 @@ get_daily_iso_checksum() mount_rootfs() { - DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +102,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-latest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +254,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +309,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -366,8 +365,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ - "${soft_link}" + ln -s "$(basename ${ROOTFS_DES_DIR})" "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 1

[PATCH compass-ci 1/2] /lib/es_query.rb: add web-backend common es methods
by Zhang Yuhang 19 Oct '20

19 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- lib/es_query.rb | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index aaaadc2..9a4e45a 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -9,12 +9,20 @@ require_relative 'constants.rb' class ESQuery HOST = (ENV.key?('ES_HOST') ? ENV['ES_HOST'] : ES_HOST) PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i - def initialize(host = HOST, port = PORT, index: 'jobs') + def initialize(host = HOST, port = PORT, index: 'jobs*') @index = index @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end + def es_query(query) + @client.search index: @index, body: query + end + + def es_count(query) + @client.count(index: @index, body: query)['count'] + end + # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) def multi_field_query(items, size: 10_000) @@ -26,7 +34,23 @@ class ESQuery } }, size: size } - @client.search index: 'jobs*', body: query + es_query(query) + end + + def all_values(field, size: 1000) + query = { + aggs: { + all_field: { + terms: { field: field, size: size } + } + }, + size: 0 + } + es_result = es_query(query)['aggregations']['all_field']['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse!.map! { |x| x['key'] } + + es_result end def query_by_id(id) -- 2.23.0

3 4

[PATCH lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 19 Oct '20

19 Oct '20

if get string named with *.yaml from ARGV, task it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..ce8accf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /^.+\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

2 1

[PATCH compass-ci] fix: sched: alter conditions when submitting cluster and normal jobs
by Ren Wen 19 Oct '20

19 Oct '20

1) when a job without 'cluster' (a job field), or has 'cluster' but starts with 'cs-localhost', the job is a normal job; otherwise, it's a cluster job. 2) alter the numbers of return values of 'get_cluster_config' function, to only return 'cluster config'. Because the conditions how to judge a job is whether a cluster job are changed. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 33 ++++++++++++--------------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4ddcec..a18bdee 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -139,22 +139,12 @@ class Sched return node_state end - # EXAMPLE: - # cluster_file: "cs-lkp-hsw-ep5" - # return: Hash(YAML::Any, YAML::Any) | Nil, 0 | <hosts_size> - # {"lkp-hsw-ep5" => {"roles" => ["server"], "macs" => ["ec:f4:bb:cb:7b:92"]}, - # "lkp-hsw-ep2" => {"roles" => ["client"], "macs" => ["ec:f4:bb:cb:54:92"]}}, 2 + # get cluster config using own lkp_src cluster file, + # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - - if File.file?(cluster_file_path) - cluster_config = YAML.parse(File.read(cluster_file_path)).as_h - hosts_size = cluster_config.values.size - return cluster_config, hosts_size - end - - return nil, 0 + return YAML.parse(File.read(cluster_file_path)).as_h end def get_commit_date(job) @@ -177,16 +167,17 @@ class Sched job = Job.new(job_content, job_content["id"]?) job["commit_date"] = get_commit_date(job) + # it is not a cluster job if cluster field is empty or + # field's prefix is 'cs-localhost' cluster_file = job["cluster"] - if cluster_file != "" - cluster_config, hosts_size = get_cluster_config( - cluster_file, job.lkp_initrd_user, job.os_arch) - - return submit_cluster_job( - job, cluster_config.not_nil!) if hosts_size >= 2 + if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + return submit_single_job(job) + else + cluster_config = get_cluster_config(cluster_file, + job.lkp_initrd_user, + job.os_arch) + return submit_cluster_job(job, cluster_config) end - - return submit_single_job(job) rescue ex puts ex.inspect_with_backtrace return [{ -- 2.23.0

2 3

[PATCH v2 compass-ci] container/assign-account: answerback-email.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add parameters for answerback-email -e,--email email_addr: add email address -s,--ssh-pubkey pub_key: add ssh pub_key -f,--raw-email mail_file: add email_adress[,pub_key] via email file why: easier for administrator to manully assign jumper account for user with: - user's email - user's email with pub_key - user's email_file Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 165 ++++++++----------- 1 file changed, 70 insertions(+), 95 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..0789c38 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -3,46 +3,18 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -=begin - -repo_list: - all repos url list from upstream-repos.git - -API: -call graph: -read_mail_content -send_account_request - email_addr - get email address - email_message_id - get email message_id - check_email_available - check email available - pub_key_value - get pub key - account - send apply account request and return account info - build_message - build email message - send_mail - call send_mail to send mail whit build message - -the returned data for account_info like: -{ - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", - "jumper_port" => "10000" -} -=end +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require 'optparse' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT CRYSTAL_INTRANET SEND_MAIL_PORT @@ -50,27 +22,71 @@ names = Set.new %w[ defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] +JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] -def build_message(email, message_id, infos) +apply_info = { + 'email_addr' => nil, + 'pub_key' => nil +} + +def format_info(email_file, apply_info) + mail_content = Mail.read(email_file) + + email_addr = mail_content.from[0] + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + apply_info['email_addr'] = email_addr + apply_info['pub_key'] = pub_key + + apply_info +end + +options = OptionParser.new do |opts| + opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key] [--raw-email email_file]\n" + opts.banner += " -e or -f is required\n" + opts.banner += ' -s is optional when use -e' + + opts.separator '' + opts.separator 'options:' + + opts.on('-e email_addr', '--email email_addr', 'appoint email address') do |email_addr| + apply_info['email_addr'] = email_addr + end + + opts.on('-s pub_key', '--ssh-pubkey pub_key', 'ssh pub_key to enable keyless login') do |pub_key| + apply_info['pub_key'] = pub_key + end + + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + format_info(email_file, apply_info) + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +options.parse!(ARGV) + +def build_message(email, acct_infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready - Dear #{email} + Dear user: Thank you for joining us. You can use the following command to login the jumper server: login command: - ssh -p #{infos['jumper_port']} #{infos['account']}@#{infos['jumper_ip']} + ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} account passwd: - account_password: #{infos['passwd']} + account_password: #{acct_infos['passwd']} regards compass-ci @@ -79,67 +95,26 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - account_info = JSON.parse account_info_str - - return account_info + account_info_str = if pub_key.nil? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end + JSON.parse account_info_str end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) +def send_account(apply_info) + message = "No email address specified\n" + message += "use -e email_addr add a email address\n" + message += 'or use -f to add a mail file' + raise message if apply_info['email_addr'].nil? - pub_key = pub_key_value(mail_content) - acct_info = account_info(pub_key) + acct_info = account_info(apply_info['pub_key']) - message = build_message(email, message_id, acct_info) + message = build_message(apply_info['email_addr'], acct_info) %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) - - return mail_content -end - -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +send_account(apply_info) -- 2.23.0

2 1

[PATCH compass-ci] job.cr: set_pp_initrd() => set_depends_initrd()
by Xu Xijian 19 Oct '20

19 Oct '20

Function set_pp_initrd() is confusing, since what it does is to set depends, so change function name to set_depends_initrd. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 20afa65..1e44d15 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -133,7 +133,7 @@ class Job set_result_root() set_result_service() set_os_mount() - set_pp_initrd() + set_depends_initrd() set_initrds_uri() set_kernel_uri() set_kernel_append_root() @@ -391,7 +391,7 @@ class Job self["user_lkp_src"] = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) end - private def set_pp_initrd + private def set_depends_initrd initrd_deps_arr = Array(String).new initrd_pkg_arr = Array(String).new initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" -- 2.23.0

3 5

[PATCH lkp-tests] /tests/build-pkg: modify save location for *.cgz file
by Liu Shaofei 19 Oct '20

19 Oct '20

1. modify *.cgz file save path /srv/initrd/build-pkg/nasm-git => /srv/initrd/build-pkg/initramfs/openeuler/aarch64/20.03/nasm-git 2. create softlink name that named by tag ../nasm-git/f71820d6055373c9b9381615e28a28f307bb2df4.CGZ ../nasm-git/1.0.4.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..65cb6dc7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,10 +23,14 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 } +create_softlink() +{ + [ -n "$upstream_tag" ] || return + [ -e "$PKG_MNT/${pack_to}/${upstream_tag}.cgz" ] && return + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" +} + check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

2 1

[PATCH compass-ci 2/2] user-client/maintain: remove job yaml redundant field
by Liu Yinsi 19 Oct '20

19 Oct '20

use submit -s 'key: value' in walk-os-iperf-test, instead of write in job yaml, remove os related parameters. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/iperf-walk-os.yaml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/user-client/maintain/walk-os-test/iperf-walk-os.yaml b/user-client/maintain/walk-os-test/iperf-walk-os.yaml index 647220c..b70b5e7 100644 --- a/user-client/maintain/walk-os-test/iperf-walk-os.yaml +++ b/user-client/maintain/walk-os-test/iperf-walk-os.yaml @@ -7,9 +7,6 @@ if role server: if role client: iperf: protocol: tcp -testbox: vm-2p8g--USER -os: OS -os_arch: OS_ARCH -os_version: OS_VER +testbox: vm-2p8g node_roles: server client LKP_DEBUG_PREFIX: bash -x -- 2.23.0

2 1

[PATCH v2 compass-ci] fix: taskqueue: simplify logs
by Tong Qunfeng 19 Oct '20

19 Oct '20

[why] current taskqueue show 4 lines for 1 debug message - empty line for split: - end user request ip:port, request body: 172.17.0.1:48956 {"domain":"..."} - response body: <-- - kemal frame log: 2020-10... 200 POST /add?queue=... 710.15µs cut it 2 line, and duplicate information - kemal frame log has time in {2020-10...} and span {710.15us} examples: before <<< 2020-10-19 02:40:06 UTC --> 172.17.0.1:48956 {"domain":"compass-ci","lab":"z9"} 2020-10-19 02:40:06 UTC <-- {"id":"z9.134780"} 2020-10-19 02:40:06 UTC 200 POST /add?queue=sched%2Ftaishan200-2280-2s64p-256g--a9 710.15µs >>> after <<< from: {172.17.0.1:37444} <-- ack: {"domain":"compass-ci","lab":"crystal","id":"crystal.104640"} 2020-10-19 08:12:20 UTC 200 PUT /consume?queue=extract_stats 1.01ms >>> Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/taskqueue/taskqueue.cr | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index c5c202d..98c5623 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -18,16 +18,22 @@ class TaskQueue @@rate_limiter.bucket(:l1pms, 1_u32, 0.001.seconds) @@rate_limiter.bucket(:l100pms, 1_u32, 0.00001.seconds) + # logs example: + # from: {172.17.0.1:6952} body {"domain":"compass-ci"} <-- ack: {"id":"z9.134780"} + # | from: {172.17.0.1:6952} <-- ack: {} + # 2020-10-19 03:25:01 UTC 201 PUT /consume?queue=sched%2Fdc-2g%2Fidle 356.51µs + # + # "from" is puts by this debug_message + # "2020-10..." auto puts by kemal frame, time in {2020-10-19 03:25:01 UTC} and span {356.51us} def debug_message(env, response, time_in) - puts("\n") + logs = "from: {#{env.request.remote_address}}" - from_message = "#{time_in} --> #{env.request.remote_address}" - if env.request.body != nil - from_message += " #{env.request.body}" - end - puts(from_message) + logs += " body: #{env.request.body}" if env.request.body + + logs += " <-- ack: " + logs += response ? "#{response}" : "{}" - puts("#{Time.utc} <-- #{response}") + puts(logs) end def run -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container/send-mail: mail-post.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add setup smpt for intranet/internet smtp enable send mail to both curystal.ci and WAN email Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/mail-post.rb | 47 ++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/container/send-mail/mail-post.rb b/container/send-mail/mail-post.rb index 4eecf6e..ffbf31d 100755 --- a/container/send-mail/mail-post.rb +++ b/container/send-mail/mail-post.rb @@ -7,23 +7,25 @@ require 'sinatra' require 'json' require 'yaml' require 'open3' -require_relative 'send-mail.rb' +require_relative 'send-mail' set :bind, '0.0.0.0' -set :port, 11311 +set :port, ENV['SEND_MAIL_PORT'] + +mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp post '/send_mail_yaml' do data = YAML.safe_load request.body.read - raise TypeError, data, 'request data type error' unless data.class.eql? Hash + raise TypeError, data, 'request data type error' unless data.instance_of?(Hash) mail_info = { 'references' => data['references'] || '', - 'from' => data['from'] || 'team(a)crystal.ci', - 'subject' => data['subject'], - 'to' => data['to'], - 'body' => data['body'] + 'from' => data['from'] || (ENV['DEFAULT_EMAIL']).to_s, + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' } - check_send_mail(mail_info) + check_send_mail(mail_info, mail_server) end post '/send_mail_text' do @@ -31,18 +33,29 @@ post '/send_mail_text' do mail_info = { 'references' => data.references || '', - 'from' => data.from || 'team(a)crystal.ci', - 'subject' => data.subject, - 'to' => data.to, - 'body' => data.body.decoded + 'from' => data.from || (ENV['DEFAULT_EMAIL']).to_s, + 'subject' => data.subject || '', + 'to' => data.to[0] || '', + 'body' => data.body.decoded || '' } - check_send_mail(mail_info) + check_send_mail(mail_info, mail_server) end -def check_send_mail(mail_info) - raise TypeError, data, 'empty subject.' if mail_info['subject'].empty? - raise TypeError, data, 'empty email address.' if mail_info['to'].empty? - raise TypeError, data, 'empty email content.' if mail_info['body'].empty? +def check_send_mail(mail_info, mail_server) + raise 'empty subject.' if mail_info['subject'].empty? + raise 'empty email address.' if mail_info['to'].empty? + raise 'empty email content.' if mail_info['body'].empty? + + check_smtp(mail_info, mail_server) send_mail(mail_info) end + +def check_smtp(mail_info, mail_server) + if [nil, 'crystal.ci'].include? mail_info['to'].split('@')[1] + intranet_smtp(mail_server) + else + internet_smtp + mail_info['from'] = (ENV['EMAIL_ADDR']).to_s + end +end -- 2.23.0

3 7

[PATCH compass-ci] service/scheduler: add ssh port to job content
by Wu Zhende 19 Oct '20

19 Oct '20

[Why] Network users need to use sshd to login testbox. So testbox should set sshd. And testbox get sshd info from job content. Scheduler needs to add ssh info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 4 ++++ src/lib/job.cr | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..1183cde 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,14 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSH_PORT_BASE + SSH_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults["SSH_PORT_BASE"] = 50000 unless defaults["SSH_PORT_BASE"] +defaults["SSH_PORT_LEN"]= 2000 unless defaults["SSH_PORT_LEN"] env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 20afa65..fcc1711 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_ssh_port() end private def append_init_field @@ -168,6 +169,11 @@ class Job end end + private def set_ssh_port + self["ssh_port_base"] = ENV["SSH_PORT_BASE"] + self["ssh_port_len"] = ENV["SSH_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

2 1

[PATCH v7 compass-ci 4/6] container/mail-robot: Dockerfile
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..57307bf --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone + +COPY mail-robot.rb /usr/local/bin/ + +COPY answerback-email.rb /usr/local/bin/ + +WORKDIR /usr/local/bin/ -- 2.23.0

2 1

[PATCH compass-ci] container/assign-account: answerback-email.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add parameters for answerback-email -e,--email email_addr: add email address -s,--ssh-pubkey pub_key: add ssh pub_key -f,--raw-email mail_file: add email_adress[,pub_key] via email file Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 162 ++++++++----------- 1 file changed, 70 insertions(+), 92 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..4a86a87 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -3,46 +3,18 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -=begin - -repo_list: - all repos url list from upstream-repos.git - -API: -call graph: -read_mail_content -send_account_request - email_addr - get email address - email_message_id - get email message_id - check_email_available - check email available - pub_key_value - get pub key - account - send apply account request and return account info - build_message - build email message - send_mail - call send_mail to send mail whit build message - -the returned data for account_info like: -{ - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", - "jumper_port" => "10000" -} -=end +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require 'optparse' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT CRYSTAL_INTRANET SEND_MAIL_PORT @@ -50,18 +22,64 @@ names = Set.new %w[ defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] +JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] -def build_message(email, message_id, infos) +email_info = { + 'email_addr' => '', + 'pub_key' => '', +} + +def format_info(email_file) + mail_content = Mail.read(email_file) + + email_addr = mail_content.from[0] + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + infos = { + 'email_addr' => email_addr, + 'pub_key' => pub_key, + } + + infos +end + +options = OptionParser.new do |opts| + opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key] [--raw-email email_file]\n" + opts.banner += ' -e or -f is required' + opts.banner += ' -s is optional when use -e' + + opts.separator '' + opts.separator 'options:' + + opts.on('-e email_addr', '--email email_addr', 'appoint email address') do |email_addr| + email_info['email_addr'] = email_addr + end + + opts.on('-s pub_key', '--ssh-pubkey pub_key', 'ssh pub_key to enable keyless login') do |pub_key| + email_info['pub_key'] = pub_key + end + + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + email_info = format_info(email_file) + end + + opts.on_tail('', '-h', '--help', 'show this message') do + puts opts + exit + end +end + +options.parse!(ARGV) + +def build_message(email, infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready - Dear #{email} + Dear #{email.split('@')[0]} Thank you for joining us. You can use the following command to login the jumper server: @@ -79,67 +97,27 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + account_info_str = if pub_key.empty? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end account_info = JSON.parse account_info_str - return account_info + account_info end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) - - pub_key = pub_key_value(mail_content) - acct_info = account_info(pub_key) +def send_account(email_info) + message = "No email address specified\n" + message += "use -e email_addr add a email address\n" + message += 'or use -f to add a mail file' + raise message if email_info['email_addr'].empty? + acct_info = account_info(email_info['pub_key']) - message = build_message(email, message_id, acct_info) + message = build_message(email_info['email_addr'], acct_info) %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) - - return mail_content -end - -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +send_account(email_info) -- 2.23.0

2 2

[PATCH compass-ci 2/2] /src/lib/web_backend.rb: remove common es methods
by Zhang Yuhang 19 Oct '20

19 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 59 +++++++++--------------------------------- 1 file changed, 12 insertions(+), 47 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..721ad77 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -30,16 +30,9 @@ NOT_SHOW_FIELDS = %w[result_root].freeze ALL_FIELDS = FIELDS + NOT_SHOW_FIELDS NOT_NEED_EXIST_FIELDS = %w[error_ids upstream_repo].freeze PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze -ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") COMPARE_RECORDS_NUMBER = 50 - -def es_query(query) - ES_CLIENT.search index: 'jobs*', body: query -end - -def es_count(query) - ES_CLIENT.count(index: 'jobs*', body: query)['count'] -end +JOBS = ESQuery.new(index: 'jobs*') +REPO = ESQuery.new(index: 'repo') # "vm-2p8g--212" remove "--212" # "vm-2p8g--zzz" remove "--zzz" @@ -47,7 +40,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end @@ -56,42 +49,15 @@ def filter_tbox_group(es_result) result.to_a end -def all_suite - query = { - aggs: { - all_suite: { - terms: { field: 'suite', size: 1000 } - } - }, - size: 0 - } - es_result = es_query(query)['aggregations']['all_suite']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - - es_result -end - def all_tbox_group - query = { - aggs: { - all_tbox_group: { - terms: { field: 'tbox_group', size: 1000 } - } - }, - size: 0 - } - es_result = es_query(query)['aggregations']['all_tbox_group']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - + es_result = JOBS.all_values('tbox_group') filter_tbox_group(es_result) end def compare_candidates_body body = { query_conditions: { - suite: all_suite, + suite: JOBS.all_values('suite'), OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, @@ -148,7 +114,7 @@ end def get_groups_matrices(conditions, dimension, must, size, from) must += build_mutli_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } - total = es_count(count_query) + total = JOBS.es_count(count_query) return {} if total < 1 query = { @@ -164,14 +130,14 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - result = es_query(query) + result = JOBS.es_query(query) matrices = combine_group_query_data(result, dimension) while matrices.empty? from += size break if from > total query[:from] = from - result = es_query(query) + result = JOBS.es_query(query) matrices = combine_group_query_data(result, dimension) end matrices @@ -229,7 +195,7 @@ def es_search(must, size, from) must << { exists: { field: f } } end count_query = { query: { bool: { must: must } } } - total = es_count(count_query) + total = JOBS.es_count(count_query) unless size size = total from = 0 @@ -245,7 +211,7 @@ def es_search(must, size, from) } return {}, total if wrong_size?(size, from) - return es_query(query)['hits']['hits'], total + return JOBS.es_query(query)['hits']['hits'], total end def get_job(result) @@ -327,7 +293,6 @@ def get_repo_url(urls) end def get_repo(git_repo) - repo = nil if git_repo must = [{ regexp: { git_repo: ".*#{git_repo}.*" } }] repo = query_repos(must, from: 0, size: 1)[0] @@ -344,7 +309,7 @@ def query_repos(must, from: 0, size: 1) git_repo: { order: 'asc' } }] } - result = ES_CLIENT.search index: 'repo', body: query + result = REPO.es_query(query) repos = [] result['hits']['hits'].each do |r| r = r['_source'] @@ -361,7 +326,7 @@ def search_repos(git_repo, page_size, page_num) from = size * page_num must = git_repo ? [{ regexp: { git_repo: ".*#{git_repo}.*" } }] : [] count_query = { query: { bool: { must: must } } } - total = ES_CLIENT.count(index: 'repo', body: count_query)['count'] + total = REPO.es_count(count_query) return [], total if wrong_size?(size, from) return query_repos(must, from: from, size: size), total -- 2.23.0

2 2

[PATCH lkp-tests 2/2] fix(iso2rootfs): support skipping iso that not used for dailybuild
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] Openeuler release two kinds of iso type now, we need to skip the released iso type that not used for dailybuild. [How] 1. Confirm whether the purpose is dailybuild by iso name. 2. Exit 0 while the released iso is not used for dailybuild. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4782378424b5..2ce4d9659f7e 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -22,6 +22,12 @@ log_info() echo "[INFO] $*" } +exit_info() +{ + log_info "$@" + exit 0 +} + die() { echo "[ERROR] $*" >&2 @@ -56,6 +62,28 @@ check_yaml_vars() done } +check_iso_name() +{ + local skipped_iso_prefixes + case ${iso_os} in + "openeuler") + skipped_iso_prefixes=( + "openEuler-2.0-SP8" + ) + ;; + *) + return + ;; + esac + + local prefix + for prefix in "${skipped_iso_prefixes[@]}" + do + [[ ${ISO_NAME} != ${prefix}* ]] || + exit_info "${iso_os} haven't release new iso for openEuler, no need to generate rootfs" + done +} + get_daily_iso_checksum() { ISO_URL="$(curl "${iso_url_file}")" @@ -63,8 +91,10 @@ get_daily_iso_checksum() local pub_ip=$(echo "${iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") ISO_URL="$(curl "${iso_url_file}" |sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" } + ISO_NAME=$(basename "$ISO_URL") ISO_CHECKSUM_URL="${ISO_URL}.sha256sum" + check_iso_name curl -o "${CHECKSUM_FILE_CACHE}.tmp" "${ISO_CHECKSUM_URL}" SHA256SUM_NET=$(awk '{print $1}' "${CHECKSUM_FILE_CACHE}.tmp") @@ -118,9 +148,8 @@ check_sha256sum_update() if [ "$SHA256SUM_CACHE" == "$SHA256SUM_NET" ] then - log_info "${iso_os} haven't release new iso, no need to generate rootfs" rm -f "${CHECKSUM_FILE_CACHE}.tmp" - exit 0 + exit_info "${iso_os} haven't release new iso, no need to generate rootfs" else log_info "${iso_os} release a new iso, start to generate rootfs ..." return -- 2.23.0

3 4

[PATCH lkp-tests 1/2] feat(iso2rootfs): modify the rootfs dir layout and rootfs dir name
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] 1. Modify the dir structure of dailybuild rootfs dir. 2. Modify the dir name of dailybuild rootfs dir. [How] before: "{os}/{os_arch}/debug-versions/dailybuild/{iso_version}-%Y%m%d%H%M%S" after: "{os}/{os_arch}/{iso_version}-%Y-%m-%d-%H-%M-%S" Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 524ab6695f93..4782378424b5 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,6 @@ get_daily_iso_checksum() mount_rootfs() { - DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +102,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +254,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +309,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -366,7 +365,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + ln -s "$(basename ${ROOTFS_DES_DIR})" \ "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH lkp-tests] tests: add test-tools
by Li Ping 19 Oct '20

19 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- tests/test-tools | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 tests/test-tools diff --git a/tests/test-tools b/tests/test-tools new file mode 100755 index 00000000..cee882aa --- /dev/null +++ b/tests/test-tools @@ -0,0 +1,37 @@ +#!/bin/bash +# - testsuite +# - case + +cd $BENCHMARK_ROOT/test-tools/mugen || exit +: "${testsuite:=smoke-testing}" +: "${case:=oe_test_dd_001}" + +set_env() +{ + bash runoet.sh -c + source ~/.bash_profile +} + +test_one_case() +{ + + bash runoet.sh -f $testsuite -r $case -x +} + +test_all_cases() +{ + bash runoet.sh -a +} + +run() +{ + set_env + if [ "${testsuite}" == "all" ] + then + test_all_cases + else + test_one_case + fi +} + +run -- 2.23.0

4 3

[PATCH v7 compass-ci 6/6] container/mail-robot: mail-robot.md
by Luan Shengde 19 Oct '20

19 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..d5eeaa9 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,45 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + - if gitee, git clone the repo and check + if non-gitee, curl url to check + - if not, raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. service log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot + +5. usage: + - mail subject: apply account + - mail body: commit url + - mail attachment: id_rsa.pub -- 2.23.0

1 0

[PATCH v7 compass-ci 3/6] container/mail-robot: build file
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container/send-mail: update start
by Luan Shengde 19 Oct '20

19 Oct '20

add setup default variables update heredoc Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 89 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 27 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..068fc1e 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,29 +1,68 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) +names = Set.new %w[ + EMAIL_PSWD + SEND_MAIL_PORT + SMTP_SERVER_ADDR + SMTP_SERVER_PORT + EMAIL_DOMAIN + EMAIL_ADDR + DEFAULT_EMAIL +] -"${cmd[@]}" +defaults = relevant_defaults(names) -cat<<EOF +EMAIL_PSWD = defaults['EMAIL_PSWD'] +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] +SMTP_SERVER_ADDR = defaults['SMTP_SERVER_ADDR'] +SMTP_SERVER_PORT = defaults['SMTP_SERVER_PORT'] +EMAIL_DOMAIN = defaults['EMAIL_DOMAIN'] +EMAIL_ADDR = defaults['EMAIL_ADDR'] +DEFAULT_EMAIL = defaults['DEFAULT_EMAIL'] + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e EMAIL_PSWD=#{EMAIL_PSWD} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -e SMTP_SERVER_ADDR=#{SMTP_SERVER_ADDR} + -e SMTP_SERVER_PORT=#{SMTP_SERVER_PORT} + -e EMAIL_DOMAIN=#{EMAIL_DOMAIN} + -e EMAIL_ADDR=#{EMAIL_ADDR} + -e DEFAULT_EMAIL=#{DEFAULT_EMAIL} + -p #{SEND_MAIL_PORT}:#{SEND_MAIL_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-mail +] + +cmd += ['bash', '-c', '/usr/local/bin/mail-post.rb'] + +system(*cmd) + +puts <<HEREDOC port: 11311 + +default from email: + when sending crystal.ci email. + team(a)crystal.ci if no from email appiontted + when sending to WAN email + compass-ci(a)qq.com in all cases + --- data='{ -"from": "email_from_addr" # default: team(a)crystal.ci +"from": "email_from_addr" "subject": "email subject", "to": "email_to_addr", "body": "email message" @@ -32,19 +71,17 @@ data='{ or data=" -from: email_from_addr # default: team(a)crystal.ci +from: email_from_addr subject: email subject to: email_to_addr body: email message" -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_yaml' -d "$data" --- data=" -From: email_from_addr # default: team(a)crystal.ci +From: email_from_addr To: email_to_addr Subject: email_subject @@ -53,8 +90,6 @@ mail_msg_line2 ... " -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_text' -d "$data" +HEREDOC -- 2.23.0

2 1

[PATCH v2 compass-ci 2/4] container/send-mail: send-mail.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add setup smtp for qq mail server enable send mail via smtp.qq.com Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..844cad5 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,15 +5,29 @@ require 'mail' -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +# mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp # setup smtp config -smtp = { - address: mail_server, - enable_starttls_auto: false -} +def intranet_smtp(mail_server) + smtp = { + address: mail_server, + enable_starttls_auto: false + } + Mail.defaults { delivery_method :smtp, smtp } +end -Mail.defaults { delivery_method :smtp, smtp } +def internet_smtp + smtp = { + address: (ENV['SMTP_SERVER_ADDR']).to_s, + port: (ENV['SMPT_SERVER_PORT']), + domain: (ENV['EMAIL_DOMAIN']).to_s, + user_name: (ENV['EMAIL_ADDR']).to_s, + password: (ENV['EMAIL_PSWD']).to_s, + openssl_verify_mode: 'none', + enable_starttls_auto: true + } + Mail.defaults { delivery_method :smtp, smtp } +end # send mail def send_mail(mail_info) -- 2.23.0

2 1

[PATCH compass-ci 1/2] user-client/maintain: test os rootfs in z9 tbox
by Liu Yinsi 19 Oct '20

19 Oct '20

1. no needed my-qemu.sh after submit job in z9 tbox, remove. 2. monitor job_state boot/extract_finished to get job runtime. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 124 +++++++++--------- 1 file changed, 65 insertions(+), 59 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..2c3fc9f 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,68 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ +# # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' +require "#{ENV['LKP_SRC']}/lib/monitor" + +TEST_OS = [ + 'openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +] + +MOUNT_TYPE = ['cifs', 'initramfs'] + +def logging(job_id, os_params, start_time, end_time, cost_time) + line = format("%-12s %-36s %-28s %-28s %-8s\n", job_id, os_params, start_time, end_time, cost_time) + f = File.new('walk-test.report', 'a') + f.write(line) + f.close +end + +def monitor_job(job_id, job_state) + res = monitor({'job_id' => job_id, 'job_state' => job_state}, {'stop' => true}, '600s') + if res == 0 + Time.now + end +end + +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) +end + +def submit_job(os_array) + os = os_array[0] + os_arch = os_array[1] + os_version = os_array[2] + os_mount = os_array[3] + message = %x(submit -s 'os: #{os}' -s 'os_arch: #{os_arch}' -s 'os_version: #{os_version}' -s 'os_mount: #{os_mount}' iperf-walk-os.yaml) + puts message + + job_id = message.split('=')[1].chomp + os_params = os_array.join('/') + start_time = monitor_job(job_id, 'boot') + end_time = monitor_job(job_id, 'extract_finished') + cost_time = (end_time - start_time).floor + + logging(job_id, os_params, start_time, end_time, cost_time) +end + +def get_os_array(os_mount) + TEST_OS.each do | item | + os_array = item.split.push(os_mount) + + submit_job(os_array) + end +end + +system "walk-test.report" + +logging("Job_ID", "OS", "Begin", "End", "Cost/sec") + +MOUNT_TYPE.each {| os_mount | get_os_array(os_mount)} -- 2.23.0

3 4

[PATCH compass-ci] fix: taskqueue: simplify logs
by Tong Qunfeng 19 Oct '20

19 Oct '20

[why] current taskqueue show 4 lines for 1 debug message - empty line for split: - end user request ip:port, request body: 172.17.0.1:48956 {"domain":"..."} - response body: <-- - kemal frame log: 2020-10... 200 POST /add?queue=... 710.15µs cut it two 2 line, and duplicate information - kemal frame log has time in {2020-10...} and span {710.15us} Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/taskqueue/taskqueue.cr | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index c5c202d..98c5623 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -18,16 +18,22 @@ class TaskQueue @@rate_limiter.bucket(:l1pms, 1_u32, 0.001.seconds) @@rate_limiter.bucket(:l100pms, 1_u32, 0.00001.seconds) + # logs example: + # from: {172.17.0.1:6952} body {"domain":"compass-ci"} <-- ack: {"id":"z9.134780"} + # | from: {172.17.0.1:6952} <-- ack: {} + # 2020-10-19 03:25:01 UTC 201 PUT /consume?queue=sched%2Fdc-2g%2Fidle 356.51µs + # + # "from" is puts by this debug_message + # "2020-10..." auto puts by kemal frame, time in {2020-10-19 03:25:01 UTC} and span {356.51us} def debug_message(env, response, time_in) - puts("\n") + logs = "from: {#{env.request.remote_address}}" - from_message = "#{time_in} --> #{env.request.remote_address}" - if env.request.body != nil - from_message += " #{env.request.body}" - end - puts(from_message) + logs += " body: #{env.request.body}" if env.request.body + + logs += " <-- ack: " + logs += response ? "#{response}" : "{}" - puts("#{Time.utc} <-- #{response}") + puts(logs) end def run -- 2.23.0

1 0

[PATCH compass-ci] add exception handing for docker.rb
by Xiao Shenwei 19 Oct '20

19 Oct '20

[why] when run multi-docker job on physical machine, run 50 client to request job, if server down or error like Connection refused. these client will exit. when the server recovers, need to run these client again. [how] add exception handing to ensure all client does not exit --- providers/docker/docker.rb | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c241d3c..2239930 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -88,6 +88,19 @@ def main(hostname) run(hostname, load_path, hash) end +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.message + sleep 10 + ensure + sleep 5 + end + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -99,10 +112,7 @@ def multi_docker(hostname, nr_container) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop do - main("#{hostname}-#{i}") - sleep 5 - end + loop_main("#{hostname}-#{i}") end pids << pid end -- 2.23.0

3 4

[PATCH v2 compass-ci 1/5] container/send-internet-mail: mail-post.rb
by Luan Shengde 18 Oct '20

18 Oct '20

send mail interface: get send mail request build email info from the request data call send-internet-mail to send email with email info Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/mail-post.rb | 46 +++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100755 container/send-internet-mail/mail-post.rb diff --git a/container/send-internet-mail/mail-post.rb b/container/send-internet-mail/mail-post.rb new file mode 100755 index 0000000..3b33c7f --- /dev/null +++ b/container/send-internet-mail/mail-post.rb @@ -0,0 +1,46 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'yaml' +require 'open3' +require_relative 'send-internet-mail.rb' + +set :bind, '0.0.0.0' +set :port, 11312 + +post '/send_mail_yaml' do + data = YAML.safe_load request.body.read + raise TypeError, data, 'request data type error' unless data.class.eql? Hash + + mail_info = { + 'references' => data['references'] || '', + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' + } + check_send_mail(mail_info) if data +end + +post '/send_mail_text' do + data = Mail.read_from_string(request.body.read) + + mail_info = { + 'references' => data.references || '', + 'subject' => data.subject || '', + 'to' => data.to || '', + 'body' => data.body.decoded || '' + } + check_send_mail(mail_info) if data +end + +def check_send_mail(mail_info) + raise 'No subject.' if mail_info['subject'].empty? + raise 'No email address.' if mail_info['to'].empty? + raise 'No email body.' if mail_info['body'].empty? + + send_mail(mail_info) +end -- 2.23.0

3 5

[PATCH lkp-tests] jobs/fio-basic-openeuler.yaml: add fio test args
by Wei Jihui 17 Oct '20

17 Oct '20

1. add disk args, it should be tested on physical machine 2. raise nr_task to 24, for numjobs is 24. 3. add direct args Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/fio-basic-openeuler.yaml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/jobs/fio-basic-openeuler.yaml b/jobs/fio-basic-openeuler.yaml index 3fcdaac8..67d4e998 100644 --- a/jobs/fio-basic-openeuler.yaml +++ b/jobs/fio-basic-openeuler.yaml @@ -3,12 +3,18 @@ suite: fio-basic category: benchmark runtime: 60 +disk: +- 1HDD +fs: +- ext4 +nr_task: 24 + fio-setup-basic: ioengine: libaio iodepth: 1024 - nr_task: 2 test_size: 1G rwmixread: 70 + direct: 1 rw: - write - read -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: load yaml config in advance
by Wei Jihui 17 Oct '20

17 Oct '20

[why] for commit c725e0f022b7: def each_jobs(&block) each_job_init - load_hosts_config job = deepcopy(@job) @job2 = {} load_defaults + load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| 1. it exec load_defaults first, so it not load hosts file when start exec load_defaults. load_defaults call expand_params, expand_params maybe expand host params, but do not have load hosts file, load_one_defaults can load host file, so exec load_one_defaults in advance. 2. load_one_defaults(file, @job) use @job is because load_hosts_file is add host config to @job, so it should be executed before "job=deepcopy(@job)" Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..27c4bd93 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -400,6 +400,11 @@ class Job @file_loaded ||= {} end + Dir.glob(["/etc/compass-ci/defaults/*.yaml", + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + load_one_defaults(file, @job) + end + i = include_files job = deepcopy(@job) revise_hash(job, deepcopy(@job2), true) @@ -418,11 +423,6 @@ class Job end @jobx = nil - Dir.glob(["/etc/compass-ci/defaults/*.yaml", - "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| - load_one_defaults(file, job) - end - for_each_in(job, i.keys) do |_pk, _h, k, v| job['___'] = v -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: add define files when program is cci-depends
by Wei Jihui 17 Oct '20

17 Oct '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..9a08dc30 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -617,9 +617,9 @@ class Job program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| - # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg + # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files - if %w(makepkg makepkg-deps pack-deps cci-makepkg).include?(program) + if %w(makepkg makepkg-deps pack-deps cci-makepkg cci-depends).include?(program) program = programs_params_hash[program]["benchmark"] end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: merge overrides to defaults
by Wei Jihui 17 Oct '20

17 Oct '20

[why] submit job.yaml testbox=vm-2p8g it should use vm-2p8g to load host file, not use defaults's testbox [slove] merge @overrides to @defaults, then use @defaults to load host file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..4e398a5a 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -382,7 +382,8 @@ class Job if defaults.is_a?(Hash) && !defaults.empty? @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) - load_hosts_config if check_load_hosts_config(defaults) + @defaults.merge!(@overrides) + load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: set flag true only when find hosts file
by Wei Jihui 17 Oct '20

17 Oct '20

it should set load_hosts_done is true only when find and load hosts file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..a5fe450f 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -307,10 +307,10 @@ class Job hwconfig = load_yaml(hosts_file, nil) @job[source_file_symkey(hosts_file)] = nil @job.merge!(hwconfig) { |_k, a, _b| a } # job's key/value has priority over hwconfig + @load_hosts_done = true end @job['arch'] ||= 'aarch64' - @load_hosts_done = true end def check_set_tbox_group -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: return the monitored logs
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] When other people use monitor to get logs, the logs only output, not return. It's hard to deal the logs. [How] Append result logs to a list. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 2a10b5c1..baf172d3 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -11,7 +11,7 @@ require 'eventmachine' require 'json' class Monitor - attr_accessor :monitor_url, :query, :overrides, :action, :job + attr_accessor :monitor_url, :query, :overrides, :action, :job, :result def initialize(monitor_url = '', query = {}, action = {}) @monitor_url = monitor_url @@ -25,6 +25,7 @@ class Monitor @exit_status_code = 0 @defaults = {} load_default + @result = [] end def load_default @@ -122,6 +123,7 @@ class Monitor ws.on :message do |event| data = JSON.parse(event.data) + @result << data output(data) connect(data, ws) -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: fix ameba warning
by Wu Zhende 17 Oct '20

17 Oct '20

parse_serial_logs.cr:46:5 [W] Lint/UselessAssign: Useless assignment to variable `host` > host = File.basename(msg["serial_path"].to_s) ^ parse_serial_logs.cr:95:5 [W] Lint/UselessAssign: Useless assignment to variable `job_id` > job_id = matched.named_captures["job_id"] ^ parse_serial_logs.cr:101:5 [W] Lint/UselessAssign: Useless assignment to variable `job` > job = Elasticsearch::Client.new.get_job_content(job_id) ^ Finished in 10.93 milliseconds 1 inspected, 3 failures. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index a000ca3..6c8a220 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -43,7 +43,7 @@ class SerialParser def host_in_msg(msg) return unless msg["serial_path"]? - host = File.basename(msg["serial_path"].to_s) + File.basename(msg["serial_path"].to_s) end def detect_start_or_end(msg, host, pattern_list) @@ -92,13 +92,13 @@ class SerialParser matched = msg["message"].to_s.match(/.*\/job_initrd_tmpfs\/(?<job_id>.*?)\//) return unless matched - job_id = matched.named_captures["job_id"] + matched.named_captures["job_id"] end def find_job(job_id) return unless job_id - job = Elasticsearch::Client.new.get_job_content(job_id) + Elasticsearch::Client.new.get_job_content(job_id) end def dump_cache(job, msg, host) -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd: forward docker logs to master-fluentd
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/fluentd/docker-fluentd.conf | 33 ++++++++++----------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index 8e76441..ac73dfb 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -60,28 +60,19 @@ </store> <store> - @type elasticsearch - host 172.17.0.1 - port 9202 - suppress_type_name true - flush_interval 1s - index_name ${tag} - ssl_verify false - num_threads 2 - </store> - - <store> - @type rabbitmq - host 172.17.0.1 - exchange logging-test - exchange_type fanout - exchange_durable false - heartbeat 10 - <format> - @type json - </format> + @type forward + flush_interval 0 + send_timeout 60 + heartbeat_type udp + heartbeat_interval 1 + recover_wait 10 + hard_timeout 60 + <server> + master-fluentd + host 172.17.0.1 + port 24225 + </server> </store> - </match> <label @FLUENT_LOG> -- 2.23.0

1 0

[PATCH lkp-tests] skip build these container for multi-docker
by Xiao Shenwei 17 Oct '20

17 Oct '20

multi-docker do't need thses container, skip it Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- setup/simplify-ci | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index e3ee3f77..eb93c150 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -36,23 +36,22 @@ boot_ipxe() run_service() { + [[ $suite == "multi-docker" ]] && return + ( cd $CCI_SRC/container/dnsmasq || return - ./build - ./start + ./build && ./start boot_ipxe )& ( cd $CCI_SRC/container/qemu-efi || return - ./build - ./install + ./build && ./install )& ( cd $CCI_SRC/container/fluentd-base || return ./build cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start + ./build && ./start )& } -- 2.23.0

1 1

[PATCH compass-ci] container/master-fluentd: add a new fluentd container
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/Dockerfile | 6 +++ container/master-fluentd/build | 6 +++ container/master-fluentd/build-depends | 1 + container/master-fluentd/docker-fluentd.conf | 48 ++++++++++++++++++++ container/master-fluentd/start | 26 +++++++++++ 5 files changed, 87 insertions(+) create mode 100644 container/master-fluentd/Dockerfile create mode 100755 container/master-fluentd/build create mode 100755 container/master-fluentd/build-depends create mode 100644 container/master-fluentd/docker-fluentd.conf create mode 100755 container/master-fluentd/start diff --git a/container/master-fluentd/Dockerfile b/container/master-fluentd/Dockerfile new file mode 100644 index 0000000..192ece3 --- /dev/null +++ b/container/master-fluentd/Dockerfile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fluentd-base:alpine + +COPY --chown=1090:1090 docker-fluentd.conf /fluentd/etc/docker-fluentd.conf diff --git a/container/master-fluentd/build b/container/master-fluentd/build new file mode 100755 index 0000000..7d0291d --- /dev/null +++ b/container/master-fluentd/build @@ -0,0 +1,6 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +system 'docker build -t master-fluentd:alpine .' diff --git a/container/master-fluentd/build-depends b/container/master-fluentd/build-depends new file mode 100755 index 0000000..29ee9d3 --- /dev/null +++ b/container/master-fluentd/build-depends @@ -0,0 +1 @@ +fluentd-base diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf new file mode 100644 index 0000000..ef97d5c --- /dev/null +++ b/container/master-fluentd/docker-fluentd.conf @@ -0,0 +1,48 @@ +<system> + workers 10 +</system> + +<source> + @type forward + bind 0.0.0.0 +</source> + +<match **> + @type copy + + <store> + @type elasticsearch + host 172.17.0.1 + port 9202 + suppress_type_name true + flush_interval 1s + index_name ${tag} + ssl_verify false + log_es_400_reason true + with_transporter_log true + </store> + + <store> + @type rabbitmq + host 172.17.0.1 + exchange logging-test + exchange_type fanout + exchange_durable false + heartbeat 10 + <format> + @type json + </format> + </store> +</match> + +<label @FLUENT_LOG> + <match fluent.*> + @type stdout + <format> + @type stdout + output_type single_value + message_key message + add_newline true + </format> + </match> +</label> diff --git a/container/master-fluentd/start b/container/master-fluentd/start new file mode 100755 index 0000000..9e02980 --- /dev/null +++ b/container/master-fluentd/start @@ -0,0 +1,26 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "master-fluentd" + +cmd = %w[ + docker run + --restart=always + --name master-fluentd + -v /etc/localtime:/etc/localtime:ro + -d + -u 1090:1090 + -p 24225:24224/tcp + -p 24225:24224/udp + -e FLUENTD_CONF=docker-fluentd.conf + --log-driver json-file + --log-opt max-size=1g + master-fluentd:alpine +] +cmd += ['sh', '-c', 'umask 002 & fluentd -c /fluentd/etc/docker-fluentd.conf'] + +system(*cmd) -- 2.23.0

1 0