October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] feat(iso2rootfs): add a soft link to the newest version of rootfs dir
by Yu Chuan 15 Oct '20

15 Oct '20

Allow job can use the debug rootfs dir by specifying 'os_version' as '{os_version}-debug'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 5184041abf2b..7ab4a08b0d60 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,13 +72,14 @@ get_daily_iso_checksum() mount_rootfs() { + DEBUG_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") - ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}/debug-versions/dailybuild" + ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" install_pkgs "nfs-utils" ;; "cifs") - ROOTFS_SERVER_PATH="//${rootfs_server}/${rootfs_path}/${iso_os}/${iso_arch}/debug-versions/dailybuild" + ROOTFS_SERVER_PATH="//${rootfs_server}/${rootfs_path}/${iso_os}/${iso_arch}" ROOTFS_MOUNT_PARAM="guest,vers=1.0,noacl,nouser_xattr" install_pkgs "cifs-utils" ;; @@ -102,7 +103,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -254,7 +255,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -309,10 +310,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -362,7 +363,12 @@ post_works() mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" - umount "${ROOTFS_LOCAL_PATH}" + cd "${ROOTFS_LOCAL_PATH}" && + rm -f "${iso_version}-debug" && + ln -s "${DEBUG_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + "${iso_version}-debug" + + cd / && umount "${ROOTFS_LOCAL_PATH}" log_info "iso2rootfs finished" } -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: find job from 2 types of tbox_group queues
by Ren Wen 15 Oct '20

15 Oct '20

Job is saved into tbox_group queues when submitting, so find job from tbox_group queues. When hostname is 'taishan200-2280-2s48p-256g--a11-12', first find job from 'taishan200-2280-2s48p-256g--a11', then find from 'taishan200-2280-2s48p-256g--a11-12'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2fbf05d..2fab207 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -417,10 +417,15 @@ class Sched end private def find_job(testbox : String, count = 1) - tbox = JobHelper.match_tbox_group(testbox) - tbox_group = tbox.partition("--")[0] - - boxes = [testbox, tbox, tbox_group] + # For example: + # testbox : taishan200-2280-2s48p-256g--a11-12 + # tbox : taishan200-2280-2s48p-256g--a11 + # tbox_group : taishan200-2280-2s48p-256g + sep = "--" + tbox = testbox.sub(/(-\d+)+$/, "") + tbox_group = tbox.rchop("#{sep}#{tbox.split(sep)[-1]}") + + boxes = [tbox, tbox_group] boxes.each do |box| count.times do job = prepare_job("sched/#{box}", testbox) -- 2.23.0

2 2

[PATCH v2 compass-ci] fix: sched: fix variable name not find
by Ren Wen 15 Oct '20

15 Oct '20

Forget to modify variable name when exchanging 2 variable names. So fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 223a30c..a4ddcec 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -418,7 +418,7 @@ class Sched private def find_job(testbox : String, count = 1) tbox_group = JobHelper.match_tbox_group(testbox) - tbox = tbox.partition("--")[0] + tbox = tbox_group.partition("--")[0] boxes = [testbox, tbox_group, tbox] boxes.each do |box| -- 2.23.0

1 0

[PATCH v3 compass-ci] fix: sched: find and submit idle job using tbox_group
by Ren Wen 15 Oct '20

15 Oct '20

Using tbox_group to find and submit idle jobs. Example: Here is a hostname: taishan200-2280-2s48p-256g--a11-<sn> Before: using taishan200-2280-2s48p-256g Now: using taishan200-2280-2s48p-256g--a11 Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2fbf05d..223a30c 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -417,10 +417,10 @@ class Sched end private def find_job(testbox : String, count = 1) - tbox = JobHelper.match_tbox_group(testbox) - tbox_group = tbox.partition("--")[0] + tbox_group = JobHelper.match_tbox_group(testbox) + tbox = tbox.partition("--")[0] - boxes = [testbox, tbox, tbox_group] + boxes = [testbox, tbox_group, tbox] boxes.each do |box| count.times do job = prepare_job("sched/#{box}", testbox) -- 2.23.0

1 1

[PATCH v7 compass-ci] sparrow: process will exit if firewalld.service was not installed
by Hu Xuejiao 15 Oct '20

15 Oct '20

When firewalld is inactive, return code of 'systemctl status firewalld' will be 3, and install-tiny has set -e, so this will cause this process exit. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..e5955d0 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -12,15 +12,16 @@ iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE -systemctl status firewalld | grep -q "running" && { +command -v firewall-cmd > /dev/null || { + exit 0 +} - DOCKER0_IFACE=docker0 - DOCKER0_SUBNET=172.17.0.0/16 +[ "$(systemctl is-active firewalld)" == "active" ] || { + exit 0 +} - iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE +DOCKER0_SUBNET=172.17.0.0/16 - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -} +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -- 2.23.0

1 0

[PATCH v2 compass-ci] sched: refactor get_boot_ipxe to job class
by Cao Xueliang 15 Oct '20

15 Oct '20

The function get_boot_ipxe use initrds_path, kernel, kernel_params from job class as response. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 120 +++++++++++++++++++++++++++++++++---- src/lib/sched.cr | 65 ++------------------ src/scheduler/constants.cr | 4 ++ 3 files changed, 116 insertions(+), 73 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 53a560c..b301c47 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -72,11 +72,15 @@ class Job lab initrd_pkg initrd_deps + initrds + initrds_path result_root access_key access_key_file lkp_initrd_user + kernel kernel_append_root + kernel_params docker_image ) @@ -123,8 +127,12 @@ class Job set_result_root() set_result_service() set_os_mount() - set_kernel_append_root() set_pp_initrd() + set_initrds() + set_initrds_path() + set_kernel() + set_kernel_append_root() + set_kernel_params() set_lkp_server() end @@ -228,27 +236,114 @@ class Job end end - private def set_kernel_append_root - os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") - lkp_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp") - lkp_basename = File.basename(lkp_real_path) + private def vmlinuz + if os_mount == "initramfs" + return "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/vmlinuz")}" + else + return "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" + end + end + + private def set_kernel + self["kernel"] = "kernel #{vmlinuz()}" + end + + private def kernel_common_params + return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" + end + + private def common_initrds + temp_initrds = [] of String + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz")}" + temp_initrds << "#{SCHED_HTTP_PREFIX}/job_initrd_tmpfs/#{id}/job.cgz" + + return temp_initrds + end + + private def initramfs_initrds + temp_initrds = [] of String + + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + + temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? + temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? + + return temp_initrds + end + + private def nfs_cifs_initrds + temp_initrds = [] of String + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp")}" + + return temp_initrds + end + + private def set_initrds + temp_initrds = [] of String - current_basename = "" if "#{os_mount}" == "initramfs" - current_real_path = JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current") - current_basename = File.basename(current_real_path) + temp_initrds.concat(initramfs_initrds()) + else + temp_initrds.concat(nfs_cifs_initrds()) end + temp_initrds.concat(common_initrds()) + + self["initrds"] = temp_initrds.join(" ") + end + + private def initrds_basename + basenames = "" + + initrds.split(/ /).each do |initrd| + basenames += "initrd=#{File.basename(initrd)} " + end + + return basenames + end + + private def set_initrds_path + paths = "" + + initrds.split(/ /).each do |initrd| + paths += "initrd #{initrd}\n" + end + + self["initrds_path"] = paths + end + + private def set_kernel_append_root + os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") + fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} initrd=#{lkp_basename}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr initrd=#{lkp_basename}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 initrd=#{current_basename}", - "container" => "" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + "container" => "", } self["kernel_append_root"] = fs2root[os_mount] end + private def kernel_console + if os_arch == "x86_64" + return "console=ttyS0,115200 console=tty0" + else + return "" + end + end + + private def set_kernel_params + self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" + end + private def set_pp_initrd initrd_deps_arr = Array(String).new initrd_pkg_arr = Array(String).new @@ -270,6 +365,7 @@ class Job end end end + self["initrd_deps"] = initrd_deps_arr.join(" ") self["initrd_pkg"] = initrd_pkg_arr.join(" ") end diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 47a2e54..04c6317 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -480,69 +480,12 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - private def add_kernel_console_param(arch_tmp) - returned = "" - if arch_tmp == "x86_64" - returned = " console=ttyS0,115200 console=tty0" - end - return returned - end - - private def get_pp_initrd(job : Job) - initrd_deps = "" - initrd_pkg = "" - if job.os_mount == "initramfs" - initrd_deps += job.initrd_deps.split.join() { |item| "initrd #{item}\n" } - initrd_pkg += job.initrd_pkg.split.join() { |item| "initrd #{item}\n" } - end - return initrd_deps, initrd_pkg - end - private def get_boot_ipxe(job : Job) - initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" - - initrd_deps, initrd_pkg = get_pp_initrd(job) - - initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" - sched_http_prefix = "http://#{SCHED_HOST}:#{SCHED_PORT}" - os_http_prefix = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" - response = "#!ipxe\n\n" - if job.os_mount == "initramfs" - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/current")}\n" - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/run-ipconfig.cgz")}\n" - else - response += "initrd #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp")}\n" - end - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}")}\n" - response += "initrd #{sched_http_prefix}/job_initrd_tmpfs/#{job.id}/job.cgz\n" - response += initrd_deps - response += initrd_pkg - if job.os_mount == "initramfs" - response += "kernel #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" - else - response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" - end - response += " user=lkp" - response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" - response += " #{job.kernel_append_root}" - response += add_kernel_console_param(job.os_arch) - if job.os_mount == "initramfs" - response += " initrd=#{initrd_lkp_cgz} initrd=job.cgz" - job.initrd_deps.split.each do |initrd_dep| - response += " initrd=#{File.basename(initrd_dep)}" - end - response += " initrd=#{File.basename(JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/run-ipconfig.cgz"))}\n" - else - response += " initrd=#{initrd_lkp_cgz} initrd=job.cgz\n" - end - response += "boot\n" + response += job.initrds_path + response += job.kernel + response += job.kernel_params + response += "\nboot\n" puts %({"job_id": "#{job.id}", "job_state": "boot"}) return response diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 7ce49a1..839c241 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,4 +24,8 @@ OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i3 SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" +OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" +SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" + DEMO_JOB = %({"suite":"pixz","testcase":"pixz","category":"benchmark","nr_threads":1,"pixz":null,"job_origin":"jobs/pixz.yaml","testbox":"wfg-e595","arch":"x86_64","tbox_group":"wfg-e595","id":"100","kmsg":null,"boot-time":null,"uptime":null,"iostat":null,"heartbeat":null,"vmstat":null,"numa-numastat":null,"numa-vmstat":null,"numa-meminfo":null,"proc-vmstat":null,"proc-stat":null,"meminfo":null,"slabinfo":null,"interrupts":null,"kconfig":"x86_64-rhel-7.6","compiler":"gcc-7"}) -- 2.23.0

2 3

[PATCH v2 compass-ci 2/2] container/assign-account: answerback-email.rb
by Luan Shengde 15 Oct '20

15 Oct '20

user email_addr,pub_key as parameter for answerback-email tool Usage: ./answerback-email.rb email_addr [pub_key] add pub_key if needed Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 78 ++++++-------------- 1 file changed, 21 insertions(+), 57 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..ad24fcd 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -34,31 +34,37 @@ the returned data for account_info like: "jumper_ip" => "10.10.10.10", "jumper_port" => "10000" } + +Usage: ruby answerback-email.rb email_addr pub_key + =end require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT CRYSTAL_INTRANET SEND_MAIL_PORT ] +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) + defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] +JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] -def build_message(email, message_id, infos) +def build_message(email, infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready Dear #{email} @@ -79,67 +85,25 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + account_info_str = if pub_key.nil? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end account_info = JSON.parse account_info_str return account_info end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) - - pub_key = pub_key_value(mail_content) +def send_account(email, pub_key) acct_info = account_info(pub_key) - message = build_message(email, message_id, acct_info) + message = build_message(email, acct_info) %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) - - return mail_content -end - -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +email = ARGV[0] +pub_key = ARGV[1] || nil +send_account(email, pub_key) -- 2.23.0

2 1

[PATCH lkp-tests 1/3] lib/job.rb: check pp key
by Wei Jihui 15 Oct '20

15 Oct '20

next when check key is nil, do not add it to pp Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 6497f0f1..8a6d68a5 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -568,14 +568,14 @@ class Job init_program_options() for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, p_args| k = hash_key_re_string(@job, p_n, @job['pp'].keys) - next if k.nil? - - unless @job[k].is_a?(Hash) + if k && !(a)job[k].is_a?(Hash) @job['pp'][k] = @job[k] next end options_hash = @referenced_programs[p_n] program_name = hash_key_re_string(h, p_n, @job['pp'].keys) + next if program_name.nil? + @job["pp"][program_name] = Hash.new() options_array = options_hash.keys options_array.each do |option| -- 2.23.0

2 1

[PATCH v2 compass-ci 1/2] container/assign-account: get_account_info.rb fix bugs
by Luan Shengde 15 Oct '20

15 Oct '20

fix bugs: unexpected errors if there is no pub_key /usr/share/ruby/json/common.rb:156:in `parse': 765: unexpected token at 'NoMethodError: undefined method `[]' for nil:NilClass (JSON::ParserError) /usr/local/bin/get_account_info.rb:96:in `setup_jumper_account_info' /usr/local/bin/assign-account.rb:24:in `block in <main>' Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/get_account_info.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 6c211a4..2f93d5b 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -33,7 +33,7 @@ setup_jumper_account_info build_account_name read_jumper_info setup_authorized_key - + the returned data for setup_jumper_account_info like: { "account" => "guest", @@ -93,15 +93,15 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] + pub_key = @data['pub_key'] unless @data.nil? jumper_ip = jumper_info[0].chomp jumper_port = jumper_info[1].chomp account = account_info[0] - passwd = if pub_key - 'Use pub_key to login' - else + passwd = if pub_key.nil? account_info[1] + else + 'Use pub_key to login' end jumper_account_info = { 'account' => account, -- 2.23.0

1 0

[PATCH compass-ci] web-backend: add "/update_repos" api and search feature to "/get_repos"
by Zhang Yuhang 15 Oct '20

15 Oct '20

"/update_repos" will be called by git mirror service. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend-nginx/nginx.conf | 5 +- container/web-backend/web-backend | 9 ++ lib/constants.rb | 3 + src/lib/web_backend.rb | 138 ++++++++++++++----------- 4 files changed, 94 insertions(+), 61 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index 5ab1a26..b461a2c 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -6,7 +6,7 @@ events { http { include /etc/nginx/mime.types; default_type application/octet-stream; - include /etc/nginx/conf.d/*.conf; + include /etc/nginx/conf.d/*.conf; server { listen 11320 ssl; server_name api.compass-ci.openeuler.org; @@ -15,8 +15,7 @@ http { ssl_session_timeout 5m; ssl_ciphers BCDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; ssl_prefer_server_ciphers on; - - location / { + location ~ ^/(get_repos|get_jobs|compare|compare_candidates)$ { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forward-For $proxy_add_x_forwarded_for; diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 60c7827..05f7f6a 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,6 +9,7 @@ require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' set :port, 32767 set :show_exceptions, false +set :environment, :production # GET /compare_candidates # return: {"query_conditions": {...}, "dimension": [...]} @@ -62,3 +63,11 @@ end get '/get_repos' do get_repos(params) end + +# POST /update_repos +# Body: repos hash json +# Content-Type: "Application/json" +# Description: update repos by git mirror service +post '/update_repos' do + update_repos(request) +end diff --git a/lib/constants.rb b/lib/constants.rb index 3a4f083..efecaef 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -10,3 +10,6 @@ ES_PORT = config['ES_PORT'] || 9200 MAIL_HOST = config['MAIL_HOST'] || '172.17.0.1' MAIL_PORT = config['MAIL_PORT'] || 11_311 + +# web-backend service port that makes git mirror service updates "repos" of web-backend service +WEB_BACKEND_PORT = 32767 diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 4544690..da1da25 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,7 +12,6 @@ require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' - FIELDS = %w[ upstream_repo os @@ -33,6 +32,66 @@ PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") COMPARE_RECORDS_NUMBER = 50 +# Repos operation class +class Repos + attr_reader :repos, :repos_list + + def initialize + @repos = {} + @repos_list = [] + + Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort.each do |repo_file| + git_repo = repo_file.slice(UPSTREAM_REPOS_PATH.size + 1) + urls = YAML.load_file(repo_file)['url'] if File.file? repo_file + repo = { + git_repo: git_repo, + git_url: get_repo_url(git_repo, urls: urls) + } + @repos_list << repo + end + end + + def get_repo_url(git_repo, urls: nil) + urls ||= @repos[git_repo]['url'] + + return if urls.nil? || urls.empty? + return urls[1] if urls[1] && urls[1][0, 4] == 'http' + + return urls[0] + end + + def find_repos(git_repo, page_num, page_size) + from = page_num * page_size + return [] if wrong_size?(page_size, from) + + repos = [] + if git_repo + @repos_list.each do |repo| + repos << repo if repo[:git_repo].include?(git_repo) + end + end + repos = @repos_list if repos.empty? + repos[from, page_size] + end + + def get_repo(git_repo) + { + git_repo: git_repo, + git_url: get_repo_url(git_repo) + } + end + + def repos=(repos) + repos_list = [] + @repos = repos + @repos.keys.sort.each do |git_repo| + repos_list << get_repo(git_repo) + end + @repos_list = repos_list + end +end + +REPOS = Repos.new def es_query(query) ES_CLIENT.search index: 'jobs*', body: query end @@ -95,7 +154,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], @@ -270,7 +329,7 @@ end def get_banner(git_repo, branches) { repo: git_repo, - git_url: get_repo(git_repo)[:git_url], + git_url: REPOS.get_repo_url(git_repo), upstream_branch: branches } end @@ -309,79 +368,42 @@ end def get_jobs(params) begin - body = get_jobs_body(params) + body = get_jobs_body(params).to_json rescue StandardError => e warn e.message return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get jobs error'] end - [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] -end - -def get_repo_url(repo_file) - return unless File.file? repo_file - - urls = YAML.load_file(repo_file)['url'] - urls.each do |url| - return url if url[0, 4] == 'http' - end - urls[0] -end - -def get_repo(git_repo, repo_file = nil) - repo = { - git_repo: git_repo, - git_url: nil - } - return repo if !git_repo && !repo_file - - if !git_repo - git_repo = repo_file[UPSTREAM_REPOS_PATH.size + 1, repo_file.size - 1] - repo[:git_repo] = git_repo - elsif !repo_file - repo_file = File.join(UPSTREAM_REPOS_PATH, git_repo) - end - - repo[:git_url] = get_repo_url(repo_file) - repo -end - -def repo_files_list - Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort -end - -def get_repos_list(repo_files, from, finish, total) - repos_list = [] - - total.times do |index| - next if index < from - break if index >= finish - - repos_list << get_repo(nil, repo_files[index]) - end - repos_list + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end def get_repos_body(params) - repo_files = repo_files_list + git_repo = space_to_nil(params[:repo]) page_size = get_positive_number(params[:page_size], 20) page_num = get_positive_number(params[:page_num], 1) - 1 - from = page_num * page_size - finish = from + page_size - total = repo_files.size - { - total: total, - repos: get_repos_list(repo_files, from, finish, total) + total: REPOS.repos_list.size, + repos: REPOS.find_repos(git_repo, page_num, page_size) } end def get_repos(params) begin - body = get_repos_body(params) + body = get_repos_body(params).to_json rescue StandardError => e warn e.message return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] end - [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def update_repos(request) + begin + request.body.rewind # in case someone already read it + REPOS.repos = JSON.parse(request.body.read) + rescue StandardError => e + warn e.message + return [500, headers, 'error'] + end + [200, headers, 'success'] end -- 2.23.0

2 2