- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] src/lib/job.cr: fix init id syntax
by Lu Kaiyi 13 Oct '20

13 Oct '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index be3446e..907644b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -50,7 +50,7 @@ class Job @hash = job_content.as_h # init job with "-1", or use the original job_content["id"] - id == "-1" if "#{id}" == "" + id = "-1" if "#{id}" == "" if initialized? return if @hash["id"] == "#{id}" -- 2.23.0

1 0

[PATCH lkp-tests] lmbench3: add lmbench.patch to PKGBUILD
by Bai Jing 13 Oct '20

13 Oct '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/lmbench3/PKGBUILD | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pkg/lmbench3/PKGBUILD b/pkg/lmbench3/PKGBUILD index 82667fcb..e1c39fcb 100644 --- a/pkg/lmbench3/PKGBUILD +++ b/pkg/lmbench3/PKGBUILD @@ -5,8 +5,10 @@ pkgdesc="A suite of simple, portable, ANSI/C microbenchmarks for UNIX/POSIX, mea arch=('i386' 'x86_64') url="http://www.bitmover.com/lmbench/" license=('GPL') -source=("https://github.com/intel/lmbench.git") -md5sums=('SKIP') +source=("https://github.com/intel/lmbench.git" + "lmbench.patch" +) +md5sums=('SKIP' 'SKIP') build() { cd "$srcdir/$pkgname" -- 2.23.0

1 0

[PATCH compass-ci 2/2] feat: The link for one command to build centos initramfs image
by Wang Chenglong 13 Oct '20

13 Oct '20

This link is used to one command to build centos-$version initramfs image Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- rootfs/initramfs/centos/aarch64/7/build | 1 + rootfs/initramfs/centos/aarch64/8/build | 1 + 2 files changed, 2 insertions(+) create mode 120000 rootfs/initramfs/centos/aarch64/7/build create mode 120000 rootfs/initramfs/centos/aarch64/8/build diff --git a/rootfs/initramfs/centos/aarch64/7/build b/rootfs/initramfs/centos/aarch64/7/build new file mode 120000 index 0000000..2d22759 --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/build @@ -0,0 +1 @@ +../../../../../container/osimage/centos/run \ No newline at end of file diff --git a/rootfs/initramfs/centos/aarch64/8/build b/rootfs/initramfs/centos/aarch64/8/build new file mode 120000 index 0000000..2d22759 --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/8/build @@ -0,0 +1 @@ +../../../../../container/osimage/centos/run \ No newline at end of file -- 2.23.0

1 0

[PATCH compass-ci] web-backend: add "/update_repos" api to update repos
by Zhang Yuhang 13 Oct '20

13 Oct '20

This will be called by git mirror service. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 8 +++ lib/constants.rb | 3 + src/lib/web_backend.rb | 107 ++++++++++++++++-------------- 3 files changed, 68 insertions(+), 50 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 60c7827..07c2f69 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -62,3 +62,11 @@ end get '/get_repos' do get_repos(params) end + +# POST /update_repos +# Body: repos hash json +# Content-Type: "Application/json" +# Description: update repos by git mirror service +post '/update_repos' do + update_repos(request) +end diff --git a/lib/constants.rb b/lib/constants.rb index 3a4f083..efecaef 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -10,3 +10,6 @@ ES_PORT = config['ES_PORT'] || 9200 MAIL_HOST = config['MAIL_HOST'] || '172.17.0.1' MAIL_PORT = config['MAIL_PORT'] || 11_311 + +# web-backend service port that makes git mirror service updates "repos" of web-backend service +WEB_BACKEND_PORT = 32767 diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 4544690..6ba18e2 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,7 +12,6 @@ require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' - FIELDS = %w[ upstream_repo os @@ -33,6 +32,48 @@ PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") COMPARE_RECORDS_NUMBER = 50 +# Repos operation class +class Repos + attr_reader :repos, :repos_list + + def initialize + @repos = {} + @repos_list = [] + + Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort.each do |repo_file| + git_repo = repo_file[UPSTREAM_REPOS_PATH.size + 1, repo_file.size - 1] + urls = YAML.load_file(repo_file)['url'] if File.file? repo_file + repo = { + git_repo: git_repo, + git_url: get_repo_url(git_repo, urls: urls) + } + @repos_list << repo + end + end + + def get_repo_url(git_repo, urls: nil) + urls ||= @repos[git_repo]['url'] + + return if urls.nil? || urls.empty? + return urls[1] if urls[1] && urls[1][0, 4] == 'http' + + return urls[0] + end + + def repos=(repos) + repos_list = [] + @repos = repos + @repos.keys.sort.each do |git_repo| + repos_list << { + git_repo: git_repo, + git_url: get_repo_url(git_repo) + } + end + @repos_list = repos_list + end +end + +REPOS = Repos.new def es_query(query) ES_CLIENT.search index: 'jobs*', body: query end @@ -95,7 +136,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], @@ -270,7 +311,7 @@ end def get_banner(git_repo, branches) { repo: git_repo, - git_url: get_repo(git_repo)[:git_url], + git_url: REPOS.get_repo_url(git_repo), upstream_branch: branches } end @@ -317,62 +358,17 @@ def get_jobs(params) [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] end -def get_repo_url(repo_file) - return unless File.file? repo_file - - urls = YAML.load_file(repo_file)['url'] - urls.each do |url| - return url if url[0, 4] == 'http' - end - urls[0] -end - -def get_repo(git_repo, repo_file = nil) - repo = { - git_repo: git_repo, - git_url: nil - } - return repo if !git_repo && !repo_file - - if !git_repo - git_repo = repo_file[UPSTREAM_REPOS_PATH.size + 1, repo_file.size - 1] - repo[:git_repo] = git_repo - elsif !repo_file - repo_file = File.join(UPSTREAM_REPOS_PATH, git_repo) - end - - repo[:git_url] = get_repo_url(repo_file) - repo -end - -def repo_files_list - Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort -end - -def get_repos_list(repo_files, from, finish, total) - repos_list = [] - - total.times do |index| - next if index < from - break if index >= finish - - repos_list << get_repo(nil, repo_files[index]) - end - repos_list -end - def get_repos_body(params) - repo_files = repo_files_list page_size = get_positive_number(params[:page_size], 20) page_num = get_positive_number(params[:page_num], 1) - 1 from = page_num * page_size finish = from + page_size - total = repo_files.size + total = REPOS.repos_list.size { total: total, - repos: get_repos_list(repo_files, from, finish, total) + repos: REPOS.repos_list[from, finish] } end @@ -385,3 +381,14 @@ def get_repos(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] end + +def update_repos(request) + begin + request.body.rewind # in case someone already read it + REPOS.repos = JSON.parse(request.body.read) + rescue StandardError => e + warn e.message + return [500, headers, 'error'] + end + [200, headers, 'success'] +end -- 2.23.0

2 2

[PATCH compass-ci] fix rubocop warnings on bisect .rb
by Cao Xueliang 13 Oct '20

13 Oct '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- lib/mail_bisect_result.rb | 4 ++-- sbin/git-bisect | 9 ++++----- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index b4bbace..d9e5cf4 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -5,7 +5,7 @@ require 'json' require_relative 'es_query' require_relative 'sched_client' -require_relative "../src/delimiter/utils" +require_relative '../src/delimiter/utils' require_relative "#{ENV['LKP_SRC']}/lib/monitor" # find the first bad commit diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 8f1a0c0..1e85824 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -36,9 +36,9 @@ class MailBisectResult https://gitee.com/openeuler/compass-ci BODY - to = "caoxl(a)crystal.ci" + to = 'caoxl(a)crystal.ci' @hash = { 'to' => to, 'body' => body, 'subject' => subject, 'from' => 'compass-ci(a)outlook.com' } - #@hash = { 'to' => @git_commit.author_email, 'body' => body, 'subject' => subject } + # @hash = { 'to' => @git_commit.author_email, 'body' => body, 'subject' => subject } end def send_mail diff --git a/sbin/git-bisect b/sbin/git-bisect index a7f158e..4744a02 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -1,6 +1,7 @@ #!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # frozen_string_literal: true + require_relative '../container/defconfig.rb' set_local_env require 'optparse' @@ -20,15 +21,13 @@ def parse_argv items end -options = {} - op = OptionParser.new do |opts| opts.banner = 'Usage: git-bisect job_id=$job_id, error_id=$error_id' - opts.separator'' + opts.separator '' opts.separator 'job_id: you submit a job to compass-ci and the job has some errors' opts.separator 'error_id: you want git bisect a error from the errors that you got' - opts.separator'' + opts.separator '' opts.on_tail('-h', '--help', 'show the help message') do puts opts @@ -44,6 +43,6 @@ unless items['job_id'] || items['error_id'] end gb = GitBisect.new(items) -result = gb.find_first_bad_commit() +result = gb.find_first_bad_commit mbr = MailBisectResult.new result mbr.create_send_email -- 2.23.0

1 0

[PATCH lkp-tests] jobs/fio-basic-openeuler.yaml: del unwanted test args
by Wei Jihui 13 Oct '20

13 Oct '20

fio test no need input args: zero_buffers,iodepth_batch, iodepth_batch_complete Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/fio-basic-openeuler.yaml | 3 --- setup/fio-setup-basic | 16 ---------------- 2 files changed, 19 deletions(-) diff --git a/jobs/fio-basic-openeuler.yaml b/jobs/fio-basic-openeuler.yaml index 7aae3eb9..3fcdaac8 100644 --- a/jobs/fio-basic-openeuler.yaml +++ b/jobs/fio-basic-openeuler.yaml @@ -8,9 +8,6 @@ fio-setup-basic: iodepth: 1024 nr_task: 2 test_size: 1G - zero_buffers: 1 - iodepth_batch: 2 - iodepth_batch_complete: 4 rwmixread: 70 rw: - write diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 157947bb..103f84ab 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -20,11 +20,7 @@ # - numa_cpu_nodes # - pre_read # - rwmixread -# - zero_buffers # - ioscheduler -# - iodepth_batch -# - iodepth_batch_complate -# - partitions . $LKP_SRC/lib/common.sh . $LKP_SRC/lib/unit.sh @@ -50,10 +46,7 @@ size=$((test_size / nr_task)) : ${file_service_type:=roundrobin} : ${random_distribution:=random} : ${pre_read:=0} -: ${zero_buffers:=0} : ${ioscheduler:=none} -: ${iodepth_batch:=0} -: ${iodepth_batch_complete:=0} direct=$(parse_bool $direct) raw_disk=$(parse_bool $raw_disk) @@ -102,15 +95,6 @@ group_reporting create_only=1 pre_read=$pre_read " -params="zero_buffers iodepth_batch iodepth_batch_complete" -for param in $params; do - param_value="$(eval echo '$'$param)" - if [ "$param_value" != 0 ]; then - create_task="\ -$create_task -$param=$param_value" - fi -done if [ "$ioscheduler" != "none" ]; then create_task="\ -- 2.23.0

2 1

[PATCH v6 compass-ci 6/6] container/mail-robot: mail-robot.md
by Luan Shengde 13 Oct '20

13 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..e248b41 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,40 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + - if gitee, git clone the repo and check + if non-gitee, curl url to check + - if not, raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. service log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

1 0

[PATCH v6 compass-ci 4/6] container/mail-robot: Dockerfile
by Luan Shengde 13 Oct '20

13 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..57307bf --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone + +COPY mail-robot.rb /usr/local/bin/ + +COPY answerback-email.rb /usr/local/bin/ + +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH v6 compass-ci 3/6] container/mail-robot: build file
by Luan Shengde 13 Oct '20

13 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v6 compass-ci 1/6] container/mail-robot: mail-robot.rb
by Luan Shengde 13 Oct '20

13 Oct '20

mail robot for application email monitor mailbox check new added email file in new check new mail's subject return if unmatched subject call answerback-email check to apply uuid/account Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/mail-robot.rb | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..393b033 --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'answerback-email.rb' + +mail_inbox = '/Maildir/.compass-ci/new/' +mail_drafts = '/Maildir/.compass-ci/cur/' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + unless added.empty? + added.each do |mail_file| + begin + check_to_send_account(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + end + listener.start + sleep +end + +def check_to_send_account(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + subject = mail_content.subject + return unless subject =~ /apply account/i + + assign_uuid = AssignUuid.new(mail_content) + assign_uuid.send_account + FileUtils.mv(mail_file, mail_drafts) +end + +monitor_dir(mail_inbox, mail_drafts) -- 2.23.0

1 0