- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7785c2a..984b9a1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + unless hash_dup[key] == @hash[key] + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,7 +389,7 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7785c2a..a0cae68 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + unless hash_dup[key] == @hash[key] + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,9 +389,9 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}")) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end -- 2.23.0

1 0

[PATCH compass-ci] job.cr: def var osimage_dir for used multi times
by Xu Xijian 16 Oct '20

16 Oct '20

osimage_dir is used twice here, define a var fo it. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..fd6e79b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -299,10 +299,11 @@ class Job private def initramfs_initrds temp_initrds = [] of String + osimage_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

1 0

[PATCH compass-ci] job.cr: def var initrd_os_dir for used multi times
by Xu Xijian 16 Oct '20

16 Oct '20

initrd_os_dir is used twice here, define a var fo it. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..fc712ba 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -299,10 +299,11 @@ class Job private def initramfs_initrds temp_initrds = [] of String + initrd_os_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + "#{JobHelper.service_path("#{initrd_os_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + "#{JobHelper.service_path("#{initrd_os_dir}/run-ipconfig.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

2 2

[PATCH lkp-tests] fix(iso2rootfs): modify the soft link name
by Yu Chuan 16 Oct '20

16 Oct '20

'dailybuild' is closer to logic than 'debug'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 7ab4a08b0d60..524ab6695f93 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,7 @@ get_daily_iso_checksum() mount_rootfs() { - DEBUG_ROOTFS_DIR="debug-versions/dailybuild" + DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +103,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +255,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +310,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -363,10 +363,11 @@ post_works() mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" + local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && - rm -f "${iso_version}-debug" && - ln -s "${DEBUG_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ - "${iso_version}-debug" + rm -f "${soft_link}" && + ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..ffe3749 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + if "#{hash_dup[key]}" != "#{@hash[key]}" + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,7 +389,7 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end -- 2.23.0

2 2

[PATCH v8 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 16 Oct '20

16 Oct '20

Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 89 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..1f7c4126 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key # email +# uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=30000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $user_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: + login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch - Login: - ssh root@$PUB_IP - Due time: - $deadline +Regards +Compass-ci +" +} - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $user_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$uuid" ] || return 0 + user_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +get_port +report_ssh_port $port run_email -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/send-mail: update start
by Luan Shengde 16 Oct '20

16 Oct '20

add setup default variables update heredoc Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 74 +++++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 27 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..e287322 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,29 +1,53 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) +names = Set.new %w[ + EMAIL_PSWD + SEND_MAIL_PORT +] -"${cmd[@]}" +defaults = relevant_defaults(names) -cat<<EOF +EMAIL_PSWD = defaults['EMAIL_PSWD'] +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e EMAIL_PSWD=#{EMAIL_PSWD} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -p #{SEND_MAIL_PORT}:#{SEND_MAIL_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-mail +] + +cmd += ['bash', '-c', '/usr/local/bin/mail-post.rb'] + +system(*cmd) + +puts <<HEREDOC port: 11311 + +default from email: + when sending crystal.ci email. + team(a)crystal.ci if no from email appiontted + when sending to WAN email + compass-ci(a)qq.com in all cases + --- data='{ -"from": "email_from_addr" # default: team(a)crystal.ci +"from": "email_from_addr" "subject": "email subject", "to": "email_to_addr", "body": "email message" @@ -32,19 +56,17 @@ data='{ or data=" -from: email_from_addr # default: team(a)crystal.ci +from: email_from_addr subject: email subject to: email_to_addr body: email message" -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_yaml' -d "$data" --- data=" -From: email_from_addr # default: team(a)crystal.ci +From: email_from_addr To: email_to_addr Subject: email_subject @@ -53,8 +75,6 @@ mail_msg_line2 ... " -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_text' -d "$data" +HEREDOC -- 2.23.0

1 0

[PATCH compass-ci 3/4] container/send-mail: update Dockerfile
by Luan Shengde 16 Oct '20

16 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/Dockerfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 36b145c..0cf3eeb 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -11,9 +11,7 @@ RUN apt-get update && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone -EXPOSE 25 - COPY mail-post.rb /usr/local/bin/ COPY send-mail.rb /usr/local/bin/ -CMD ["/usr/local/bin/mail-post.rb"] +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/send-mail: send-mail.rb
by Luan Shengde 16 Oct '20

16 Oct '20

add setup smtp for qq.com enable send mail via smtp.qq.com Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..4fb2b7e 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,15 +5,29 @@ require 'mail' -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +# mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp # setup smtp config -smtp = { - address: mail_server, - enable_starttls_auto: false -} +def intranet_smtp(mail_server) + smtp = { + address: mail_server, + enable_starttls_auto: false + } + Mail.defaults { delivery_method :smtp, smtp } +end -Mail.defaults { delivery_method :smtp, smtp } +def internet_smtp + smtp = { + address: 'smtp.qq.com', + port: 25, + domain: 'qq.com', + user_name: 'compass-ci(a)qq.com', + password: (ENV['EMAIL_PSWD']).to_s, + openssl_verify_mode: 'none', + enable_starttls_auto: true + } + Mail.defaults { delivery_method :smtp, smtp } +end # send mail def send_mail(mail_info) -- 2.23.0

1 0