- Compass-ci - mailweb.openeuler.org

[PATCH v7 compass-ci 02/12] send-internet-mail: optimize
by Luan Shengde 14 Jan '21

14 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 1

[PATCH v7 compass-ci 01/12] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 14 Jan '21

14 Jan '21

[why] redis: used to read/store email counts from/to redis queue rufus-scheduler: used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 1

Re: [PATCH v2 lkp-tests] jobs: and ansible_test.yaml
by Liu Shaofei 14 Jan '21

14 Jan '21

^---- and => add Reply-To: In-Reply-To: <20210114021509.109179-1-ylzhangah(a)qq.com> On Thu, Jan 14, 2021 at 10:15:09AM +0800, Zhang Yale wrote: >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > jobs/ansible_test.yaml | 6 ++++++ > 1 file changed, 6 insertions(+) > create mode 100644 jobs/ansible_test.yaml > >diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml >new file mode 100644 >index 00000000..49f31562 >--- /dev/null >+++ b/jobs/ansible_test.yaml >@@ -0,0 +1,6 @@ >+suite: ansible_test >+category: functional >+ >+ansible_test: >+ ansible_repo: >+# - geerlingguy/ansible-role-jenkins ^---- whether to remove '#'? Thanks, Liushaofei >-- >2.23.0 >

2 1

[PATCH lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

output: ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 0 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..74ac32f1 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /\: ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /.*playbook_run_on_fail/ + puts line + end +end + -- 2.23.0

1 1

[PATCH compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 14 Jan '21

14 Jan '21

[Why] Sometimes, the kernel is in vmlinux format and needs to be unpacked into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 57 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 58 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index e908079..66aba62 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -133,7 +133,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +155,58 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +get_vmlinuz_start_postion() { + local vmlinuz=$1 + + local vmlinuz_start_line + local vmlinuz_start_postion + # '1f 8b 08' - https://www.filesignatures.net/index.php?page=search&search=1F8B08&mode=SIG + vmlinuz_start_line=$(od -A d -t x1 ${vmlinuz} | grep "1f 8b 08") + vmlinuz_start_postion=${vmlinuz_start_line%% *} + + [ -z ${vmlinuz_start_postion} ] && { + echo "[WARNING] identify vmlinuz failed." + export VMLINUZ_START_POSTION=-1 + return + } + + [ X${vmlinuz_start_postion} == X0000000 ] || { + local vmlinuz_start_postion_tmp + vmlinuz_start_postion_tmp=$(echo ${vmlinuz_start_line} | awk '{for(i=1;i<=NF;i++)if($i=="1f")print i-1}') + vmlinuz_start_postion=$((${vmlinuz_start_postion} + ${vmlinuz_start_postion_tmp} -1)) + } + + export VMLINUZ_START_POSTION=${vmlinuz_start_postion} +} + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + local cmds + cmds=( + cd $ROOTFS_DIR/boot + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + ) + if "${cmds[@]}" ; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

3 4

[PATCH v2 compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 14 Jan '21

14 Jan '21

[Why] Sometimes, the kernel is in vmlinux format and needs to be unpacked into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 57 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 58 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index c26bbc8..91d53e7 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -133,7 +133,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +155,58 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +get_vmlinuz_start_postion() { + local vmlinuz=$1 + + local vmlinuz_start_line + local vmlinuz_start_postion + # '1f 8b 08' - https://www.filesignatures.net/index.php?page=search&search=1F8B08&mode=SIG + vmlinuz_start_line=$(od -A d -t x1 ${vmlinuz} | grep "1f 8b 08") + vmlinuz_start_postion=${vmlinuz_start_line%% *} + + [ -z ${vmlinuz_start_postion} ] && { + echo "[WARNING] identify vmlinuz failed." + export VMLINUZ_START_POSTION=-1 + return + } + + [ X${vmlinuz_start_postion} == X0000000 ] || { + local vmlinuz_start_postion_tmp + vmlinuz_start_postion_tmp=$(echo ${vmlinuz_start_line} | awk '{for(i=1;i<=NF;i++)if($i=="1f")print i-1}') + vmlinuz_start_postion=$((${vmlinuz_start_postion} + ${vmlinuz_start_postion_tmp} -1)) + } + + export VMLINUZ_START_POSTION=${vmlinuz_start_postion} +} + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + local cmds + cmds=( + cd $ROOTFS_DIR/boot + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + ) + if "${cmds[@]}" ; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

2 1

[PATCH lkp-tests] stats: add stats script for openeuler_docker test
by Bai Jing 14 Jan '21

14 Jan '21

[input] ^[[0mwordpress_build_on_fail: 1 [output] wordpress_build_on_fail: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 stats/openeuler_docker diff --git a/stats/openeuler_docker b/stats/openeuler_docker new file mode 100755 index 000000000..5ede8a240 --- /dev/null +++ b/stats/openeuler_docker @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + line.chomp! + next unless line =~ /.*_build_on_fail/ + + puts line +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] delimiter: use assist-result container client to get job yaml
by Cao Xueliang 14 Jan '21

14 Jan '21

Before: Get the job yaml from es, we need deal some process products of job. After: Get the job yaml from assist-result container without the process products. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 2 +- src/delimiter/utils.rb | 24 +++++++----------------- 2 files changed, 8 insertions(+), 18 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index ebf90cc..7b14558 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,7 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' +DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index de5a302..61fe6ab 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -9,6 +9,7 @@ require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/assist_result_client' require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" @@ -121,9 +122,10 @@ module Utils end def init_job_content(job_id) - es = ESQuery.new - job = es.query_by_id(job_id) - raise "es query job id: #{job_id} failed!" unless job + job_yaml = AssistResult.new.get_job_yaml(job_id) + raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + + job = JSON.parse job_yaml account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info @@ -135,21 +137,9 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id - job['testbox'] = job['config'] ? 'dc-16g' : 'dc-8g' - job['os'] = 'openeuler' - job['os_arch'] = 'aarch64' - job['os_mount'] = 'container' - job['os_version'] = '20.03-pre' - job['docker_image'] = 'openeuler:20.03-pre' - - job.delete('start_time') - job.delete('tbox_group') - job.delete('error_ids') - job.delete('job_state') - job.delete('end_time') - job.delete('loadavg') + + job.delete('subqueue') job.delete('queue') - job.delete('stats') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib: add assist_result_client for assist-result container
by Cao Xueliang 14 Jan '21

14 Jan '21

[why] Add client for assist-result container, others can call the API by the client. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/assist_result_client.rb | 22 ++++++++++++++++++++++ lib/constants.rb | 3 +++ 2 files changed, 25 insertions(+) create mode 100644 lib/assist_result_client.rb diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb new file mode 100644 index 0000000..042b189 --- /dev/null +++ b/lib/assist_result_client.rb @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rest-client' +require_relative 'constants' + +class AssistResult + def initialize(host = ASSIST_RESULT_HOST, port = ASSIST_RESULT_PORT) + @host = host + @port = port + end + + def get_job_yaml(job_id) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/get_job_yaml/#{job_id}") + response = resource.get() + return nil unless response.code == 200 + + return response.body + end +end diff --git a/lib/constants.rb b/lib/constants.rb index a274adc..ffc4f03 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -15,3 +15,6 @@ SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' + +ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' +ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: and ansible_test.yaml
by Zhang Yale 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..49f31562 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,6 @@ +suite: ansible_test +category: functional + +ansible_test: + ansible_repo: +# - geerlingguy/ansible-role-jenkins -- 2.23.0

1 0