- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests: lmbench3 need to be adapted aarch64 architecture
by Bai Jing 13 Oct '20

13 Oct '20

problem : cat:results/x86_64-linux-gnu/vm-aarch64-2p8g--baijing-2994780.0: No such file or directory Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/lmbench3 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/lmbench3 b/tests/lmbench3 index 8570a35a..f09c49e2 100755 --- a/tests/lmbench3 +++ b/tests/lmbench3 @@ -138,7 +138,7 @@ run_lmbench3_development() eval "$test=yes" - sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/x86_64-linux-gnu/lmbench + sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/aarch64-linux-gnu/lmbench log_eval " ( echo $nr_threads @@ -177,7 +177,7 @@ run_lmbench3_development() show_result() { - cat results/x86_64-linux-gnu/$testbox.0 || { + cat results/aarch64-linux-gnu/$testbox.0 || { echo "no test result found" exit -1 } -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: fix init id syntax
by Lu Kaiyi 13 Oct '20

13 Oct '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index be3446e..907644b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -50,7 +50,7 @@ class Job @hash = job_content.as_h # init job with "-1", or use the original job_content["id"] - id == "-1" if "#{id}" == "" + id = "-1" if "#{id}" == "" if initialized? return if @hash["id"] == "#{id}" -- 2.23.0

1 0

[PATCH lkp-tests] lmbench3: add lmbench.patch to PKGBUILD
by Bai Jing 13 Oct '20

13 Oct '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/lmbench3/PKGBUILD | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pkg/lmbench3/PKGBUILD b/pkg/lmbench3/PKGBUILD index 82667fcb..e1c39fcb 100644 --- a/pkg/lmbench3/PKGBUILD +++ b/pkg/lmbench3/PKGBUILD @@ -5,8 +5,10 @@ pkgdesc="A suite of simple, portable, ANSI/C microbenchmarks for UNIX/POSIX, mea arch=('i386' 'x86_64') url="http://www.bitmover.com/lmbench/" license=('GPL') -source=("https://github.com/intel/lmbench.git") -md5sums=('SKIP') +source=("https://github.com/intel/lmbench.git" + "lmbench.patch" +) +md5sums=('SKIP' 'SKIP') build() { cd "$srcdir/$pkgname" -- 2.23.0

1 0

[PATCH compass-ci 2/2] feat: The link for one command to build centos initramfs image
by Wang Chenglong 13 Oct '20

13 Oct '20

This link is used to one command to build centos-$version initramfs image Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- rootfs/initramfs/centos/aarch64/7/build | 1 + rootfs/initramfs/centos/aarch64/8/build | 1 + 2 files changed, 2 insertions(+) create mode 120000 rootfs/initramfs/centos/aarch64/7/build create mode 120000 rootfs/initramfs/centos/aarch64/8/build diff --git a/rootfs/initramfs/centos/aarch64/7/build b/rootfs/initramfs/centos/aarch64/7/build new file mode 120000 index 0000000..2d22759 --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/build @@ -0,0 +1 @@ +../../../../../container/osimage/centos/run \ No newline at end of file diff --git a/rootfs/initramfs/centos/aarch64/8/build b/rootfs/initramfs/centos/aarch64/8/build new file mode 120000 index 0000000..2d22759 --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/8/build @@ -0,0 +1 @@ +../../../../../container/osimage/centos/run \ No newline at end of file -- 2.23.0

1 0

[PATCH compass-ci] web-backend: add "/update_repos" api to update repos
by Zhang Yuhang 13 Oct '20

13 Oct '20

This will be called by git mirror service. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 8 +++ lib/constants.rb | 3 + src/lib/web_backend.rb | 107 ++++++++++++++++-------------- 3 files changed, 68 insertions(+), 50 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 60c7827..07c2f69 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -62,3 +62,11 @@ end get '/get_repos' do get_repos(params) end + +# POST /update_repos +# Body: repos hash json +# Content-Type: "Application/json" +# Description: update repos by git mirror service +post '/update_repos' do + update_repos(request) +end diff --git a/lib/constants.rb b/lib/constants.rb index 3a4f083..efecaef 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -10,3 +10,6 @@ ES_PORT = config['ES_PORT'] || 9200 MAIL_HOST = config['MAIL_HOST'] || '172.17.0.1' MAIL_PORT = config['MAIL_PORT'] || 11_311 + +# web-backend service port that makes git mirror service updates "repos" of web-backend service +WEB_BACKEND_PORT = 32767 diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 4544690..6ba18e2 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,7 +12,6 @@ require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' - FIELDS = %w[ upstream_repo os @@ -33,6 +32,48 @@ PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") COMPARE_RECORDS_NUMBER = 50 +# Repos operation class +class Repos + attr_reader :repos, :repos_list + + def initialize + @repos = {} + @repos_list = [] + + Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort.each do |repo_file| + git_repo = repo_file[UPSTREAM_REPOS_PATH.size + 1, repo_file.size - 1] + urls = YAML.load_file(repo_file)['url'] if File.file? repo_file + repo = { + git_repo: git_repo, + git_url: get_repo_url(git_repo, urls: urls) + } + @repos_list << repo + end + end + + def get_repo_url(git_repo, urls: nil) + urls ||= @repos[git_repo]['url'] + + return if urls.nil? || urls.empty? + return urls[1] if urls[1] && urls[1][0, 4] == 'http' + + return urls[0] + end + + def repos=(repos) + repos_list = [] + @repos = repos + @repos.keys.sort.each do |git_repo| + repos_list << { + git_repo: git_repo, + git_url: get_repo_url(git_repo) + } + end + @repos_list = repos_list + end +end + +REPOS = Repos.new def es_query(query) ES_CLIENT.search index: 'jobs*', body: query end @@ -95,7 +136,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], @@ -270,7 +311,7 @@ end def get_banner(git_repo, branches) { repo: git_repo, - git_url: get_repo(git_repo)[:git_url], + git_url: REPOS.get_repo_url(git_repo), upstream_branch: branches } end @@ -317,62 +358,17 @@ def get_jobs(params) [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] end -def get_repo_url(repo_file) - return unless File.file? repo_file - - urls = YAML.load_file(repo_file)['url'] - urls.each do |url| - return url if url[0, 4] == 'http' - end - urls[0] -end - -def get_repo(git_repo, repo_file = nil) - repo = { - git_repo: git_repo, - git_url: nil - } - return repo if !git_repo && !repo_file - - if !git_repo - git_repo = repo_file[UPSTREAM_REPOS_PATH.size + 1, repo_file.size - 1] - repo[:git_repo] = git_repo - elsif !repo_file - repo_file = File.join(UPSTREAM_REPOS_PATH, git_repo) - end - - repo[:git_url] = get_repo_url(repo_file) - repo -end - -def repo_files_list - Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort -end - -def get_repos_list(repo_files, from, finish, total) - repos_list = [] - - total.times do |index| - next if index < from - break if index >= finish - - repos_list << get_repo(nil, repo_files[index]) - end - repos_list -end - def get_repos_body(params) - repo_files = repo_files_list page_size = get_positive_number(params[:page_size], 20) page_num = get_positive_number(params[:page_num], 1) - 1 from = page_num * page_size finish = from + page_size - total = repo_files.size + total = REPOS.repos_list.size { total: total, - repos: get_repos_list(repo_files, from, finish, total) + repos: REPOS.repos_list[from, finish] } end @@ -385,3 +381,14 @@ def get_repos(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] end + +def update_repos(request) + begin + request.body.rewind # in case someone already read it + REPOS.repos = JSON.parse(request.body.read) + rescue StandardError => e + warn e.message + return [500, headers, 'error'] + end + [200, headers, 'success'] +end -- 2.23.0

2 2

[PATCH compass-ci] fix rubocop warnings on bisect .rb
by Cao Xueliang 13 Oct '20

13 Oct '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- lib/mail_bisect_result.rb | 4 ++-- sbin/git-bisect | 9 ++++----- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index b4bbace..d9e5cf4 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -5,7 +5,7 @@ require 'json' require_relative 'es_query' require_relative 'sched_client' -require_relative "../src/delimiter/utils" +require_relative '../src/delimiter/utils' require_relative "#{ENV['LKP_SRC']}/lib/monitor" # find the first bad commit diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 8f1a0c0..1e85824 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -36,9 +36,9 @@ class MailBisectResult https://gitee.com/openeuler/compass-ci BODY - to = "caoxl(a)crystal.ci" + to = 'caoxl(a)crystal.ci' @hash = { 'to' => to, 'body' => body, 'subject' => subject, 'from' => 'compass-ci(a)outlook.com' } - #@hash = { 'to' => @git_commit.author_email, 'body' => body, 'subject' => subject } + # @hash = { 'to' => @git_commit.author_email, 'body' => body, 'subject' => subject } end def send_mail diff --git a/sbin/git-bisect b/sbin/git-bisect index a7f158e..4744a02 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -1,6 +1,7 @@ #!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # frozen_string_literal: true + require_relative '../container/defconfig.rb' set_local_env require 'optparse' @@ -20,15 +21,13 @@ def parse_argv items end -options = {} - op = OptionParser.new do |opts| opts.banner = 'Usage: git-bisect job_id=$job_id, error_id=$error_id' - opts.separator'' + opts.separator '' opts.separator 'job_id: you submit a job to compass-ci and the job has some errors' opts.separator 'error_id: you want git bisect a error from the errors that you got' - opts.separator'' + opts.separator '' opts.on_tail('-h', '--help', 'show the help message') do puts opts @@ -44,6 +43,6 @@ unless items['job_id'] || items['error_id'] end gb = GitBisect.new(items) -result = gb.find_first_bad_commit() +result = gb.find_first_bad_commit mbr = MailBisectResult.new result mbr.create_send_email -- 2.23.0

1 0

[PATCH lkp-tests] jobs/fio-basic-openeuler.yaml: del unwanted test args
by Wei Jihui 13 Oct '20

13 Oct '20

fio test no need input args: zero_buffers,iodepth_batch, iodepth_batch_complete Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/fio-basic-openeuler.yaml | 3 --- setup/fio-setup-basic | 16 ---------------- 2 files changed, 19 deletions(-) diff --git a/jobs/fio-basic-openeuler.yaml b/jobs/fio-basic-openeuler.yaml index 7aae3eb9..3fcdaac8 100644 --- a/jobs/fio-basic-openeuler.yaml +++ b/jobs/fio-basic-openeuler.yaml @@ -8,9 +8,6 @@ fio-setup-basic: iodepth: 1024 nr_task: 2 test_size: 1G - zero_buffers: 1 - iodepth_batch: 2 - iodepth_batch_complete: 4 rwmixread: 70 rw: - write diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 157947bb..103f84ab 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -20,11 +20,7 @@ # - numa_cpu_nodes # - pre_read # - rwmixread -# - zero_buffers # - ioscheduler -# - iodepth_batch -# - iodepth_batch_complate -# - partitions . $LKP_SRC/lib/common.sh . $LKP_SRC/lib/unit.sh @@ -50,10 +46,7 @@ size=$((test_size / nr_task)) : ${file_service_type:=roundrobin} : ${random_distribution:=random} : ${pre_read:=0} -: ${zero_buffers:=0} : ${ioscheduler:=none} -: ${iodepth_batch:=0} -: ${iodepth_batch_complete:=0} direct=$(parse_bool $direct) raw_disk=$(parse_bool $raw_disk) @@ -102,15 +95,6 @@ group_reporting create_only=1 pre_read=$pre_read " -params="zero_buffers iodepth_batch iodepth_batch_complete" -for param in $params; do - param_value="$(eval echo '$'$param)" - if [ "$param_value" != 0 ]; then - create_task="\ -$create_task -$param=$param_value" - fi -done if [ "$ioscheduler" != "none" ]; then create_task="\ -- 2.23.0

2 1

[PATCH v6 compass-ci 6/6] container/mail-robot: mail-robot.md
by Luan Shengde 13 Oct '20

13 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..e248b41 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,40 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + - if gitee, git clone the repo and check + if non-gitee, curl url to check + - if not, raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. service log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

1 0

[PATCH v6 compass-ci 4/6] container/mail-robot: Dockerfile
by Luan Shengde 13 Oct '20

13 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..57307bf --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone + +COPY mail-robot.rb /usr/local/bin/ + +COPY answerback-email.rb /usr/local/bin/ + +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH v6 compass-ci 3/6] container/mail-robot: build file
by Luan Shengde 13 Oct '20

13 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0