- Compass-ci - mailweb.openeuler.org

[PATCH v9 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 19 Oct '20

19 Oct '20

Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 89 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..55015b40 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key # email +# my_uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=30000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $my_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch - Login: - ssh root@$PUB_IP - Due time: - $deadline +Regards +Compass-ci +" +} + +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +Dear $my_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +set_port() +{ + [ -n "$my_uuid" ] || return + + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] && return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$my_uuid" ] || return 0 + my_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +set_port +report_ssh_port $port run_email -- 2.23.0

3 3

[PATCH compass-ci] job.cr: initramfs and nfs use the same kernel file
by Xu Xijian 19 Oct '20

19 Oct '20

Actually the vmlinuz file in SRV_INITRD is copy from SRV_OS, so they are the same file, just use the same path is enough. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..fc15441 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -272,18 +272,9 @@ class Job return true end - private def vmlinuz - if os_mount == "initramfs" - return "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/vmlinuz")}" - else - return "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" - end - end - private def set_kernel_uri - self["kernel_uri"] = "kernel #{vmlinuz()}" + self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" end private def kernel_common_params -- 2.23.0

1 0

[PATCH lab-z9] allot/idle: remove the field -hi1620 from tbox_group
by Li Ping 19 Oct '20

19 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml index 4525fcd..513b631 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml @@ -8,11 +8,11 @@ os_mount: initramfs simplify-ci: multi-qemu-0: nr_vm: 20 - tbox_group: vm-hi1620-2p8g + tbox_group: vm-2p8g multi-qemu-1: nr_vm: 2 - tbox_group: vm-hi1620-2p4g + tbox_group: vm-2p4g multi-qemu-2: nr_vm: 2 - tbox_group: vm-hi1620-1p1g + tbox_group: vm-1p1g sleep: 100d -- 2.23.0

1 0

[PATCH v7 compass-ci 1/6] container/mail-robot: mail-robot.rb
by Luan Shengde 19 Oct '20

19 Oct '20

mail robot for email application monitor mailbox check new added email file in new check new mail's subject return if unmatched subject call answerback-email check to apply uuid/account Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/mail-robot.rb | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..70b8b25 --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'answerback-email.rb' + +mail_inbox = '/srv/cci/Maildir/.compass-ci/new/' +mail_drafts = '/srv/cci/Maildir/.compass-ci/cur/' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + unless added.empty? + added.each do |mail_file| + begin + check_to_send_account(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + end + listener.start + sleep +end + +def check_to_send_account(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + subject = mail_content.subject + return unless subject =~ /apply account/i + + assign_uuid = AssignUuid.new(mail_content) + assign_uuid.send_account + FileUtils.mv(mail_file, mail_drafts) +end + +monitor_dir(mail_inbox, mail_drafts) -- 2.23.0

3 4

[PATCH lkp-tests] lib/job.rb: load hosts file for atomic job
by Wei Jihui 19 Oct '20

19 Oct '20

[why] for commit c725e0f022b7: def each_jobs(&block) each_job_init - load_hosts_config job = deepcopy(@job) @job2 = {} load_defaults + load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| job = deepcopy(@job) this @job has no load hosts config, so it should do it under "@jobs.each do |hash|" for atomic job. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..6359254e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -696,9 +696,11 @@ class Job each_job_init each_job(&block) @jobs.each do |hash| + @load_hosts_done = false @job = deepcopy(job) @job2 = hash load_defaults + load_hosts_config each_job_init each_job(&block) end -- 2.23.0

2 2

[PATCH v7 compass-ci 5/6] container/mail-robot: start file
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/start | 39 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..108ed83 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_INTERNET_MAIL_PORT +] + +defaults = relevant_defaults(names) + +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] +SEND_MAIL_PORT = defaults['SEND_INTERNET_MAIL_PORT'] + +cmd = %W[ + docker run + --restart=always + --name=mail-robot + -d + -e J_HOST=#{JUMPER_HOST} + -e J_PORT=#{JUMPER_PORT} + -e S_M_PORT=#{SEND_MAIL_PORT} + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir:/srv/cci/Maildir:rw + -v /c/compass-ci:/c/compass-ci:ro + mail-robot +] + +cmd += ['bash', '-c', 'umask 002 && ruby ./mail-robot.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v2 compass-ci] sched: throw an exception when deps/pkg does not exist
by Cao Xueliang 19 Oct '20

19 Oct '20

When submitting a job, an exception is thrown when deps/pkg does not exist on the server. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..7785c2a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -389,10 +389,12 @@ class Job program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") && File.exists?(pkg_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

2 5

[PATCH v2 lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 19 Oct '20

19 Oct '20

if get string named with *.yaml from ARGV, treat it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..ce8accf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /^.+\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/2] feat(iso2rootfs): modify the rootfs dir layout and rootfs dir name
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] 1. Modify the dir structure of dailybuild rootfs dir. 2. Modify the dir name of dailybuild rootfs dir. [How] before: "{os}/{os_arch}/debug-versions/dailybuild/{iso_version}-%Y%m%d%H%M%S" after: "{os}/{os_arch}/{iso_version}-%Y-%m-%d-%H-%M-%S" Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 524ab6695f93..89328f2f4a9b 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,6 @@ get_daily_iso_checksum() mount_rootfs() { - DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +102,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-latest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +254,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +309,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -366,8 +365,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ - "${soft_link}" + ln -s "$(basename ${ROOTFS_DES_DIR})" "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 1

[PATCH compass-ci 1/2] /lib/es_query.rb: add web-backend common es methods
by Zhang Yuhang 19 Oct '20

19 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- lib/es_query.rb | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index aaaadc2..9a4e45a 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -9,12 +9,20 @@ require_relative 'constants.rb' class ESQuery HOST = (ENV.key?('ES_HOST') ? ENV['ES_HOST'] : ES_HOST) PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i - def initialize(host = HOST, port = PORT, index: 'jobs') + def initialize(host = HOST, port = PORT, index: 'jobs*') @index = index @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end + def es_query(query) + @client.search index: @index, body: query + end + + def es_count(query) + @client.count(index: @index, body: query)['count'] + end + # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) def multi_field_query(items, size: 10_000) @@ -26,7 +34,23 @@ class ESQuery } }, size: size } - @client.search index: 'jobs*', body: query + es_query(query) + end + + def all_values(field, size: 1000) + query = { + aggs: { + all_field: { + terms: { field: field, size: size } + } + }, + size: 0 + } + es_result = es_query(query)['aggregations']['all_field']['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse!.map! { |x| x['key'] } + + es_result end def query_by_id(id) -- 2.23.0

3 4