January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] doc: add a doc about catalogs under /srv/git
by Li Yuanchao 13 Jan '21

13 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/about_srv_git.md | 182 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 doc/about_srv_git.md diff --git a/doc/about_srv_git.md b/doc/about_srv_git.md new file mode 100644 index 0000000..3374828 --- /dev/null +++ b/doc/about_srv_git.md @@ -0,0 +1,182 @@ + +# ��`/srv/git`�� + +``` +delimiter: + ��git-bisect,��good commit�� + +auto-submit�� + ��job��rabbitmq��git-mirror��job�� + +git-daemon�� + ��git��/srv/git��git clone/git push�� + �� "git clone git://localhost/$project" ��git-daemon�� + +git-mirror�� + ��clone�� + +remote-git�� + ��/srv/git��curl��/srv/git��git�� +``` + + +# /srv/git�� + + +��a-z��git-mirror�� +``` +drwxrwsr-x. 55 lkp committer 4.0K 2020-11-20 03:54 u +drwxrwsr-x. 78 lkp committer 4.0K 2020-12-04 11:24 v +drwxrwsr-x. 75 lkp committer 4.0K 2020-12-04 11:25 i +drwxrwsr-x. 986 lkp committer 36K 2020-12-04 11:37 d +drwxrwsr-x. 1612 lkp committer 60K 2020-12-05 03:50 g +drwxrwsr-x. 827 lkp committer 28K 2020-12-05 04:00 b +drwxrwsr-x. 66 lkp committer 4.0K 2020-12-05 04:05 w +drwxrwsr-x. 32 lkp committer 4.0K 2020-12-05 04:17 y +drwxrwsr-x. 706 lkp committer 28K 2020-12-05 05:49 e +drwxrwsr-x. 837 lkp committer 28K 2020-12-08 22:05 f +drwxrwsr-x. 679 lkp committer 24K 2020-12-10 18:18 k +drwxrwsr-x. 948 lkp committer 36K 2020-12-11 02:31 h +drwxrwsr-x. 384 lkp committer 12K 2020-12-11 03:12 j +drwxrwsr-x. 927 lkp committer 36K 2020-12-15 03:11 n +drwxrwsr-x. 1757 lkp committer 56K 2020-12-15 11:09 l +drwxrwsr-x. 54 lkp committer 4.0K 2020-12-15 14:29 t +drwxrwsr-x. 1018 lkp committer 36K 2020-12-15 15:57 a +drwxrwsr-x. 1344 lkp committer 40K 2020-12-15 22:28 m +drwxrwsr-x. 809 lkp committer 28K 2020-12-16 05:21 o +drwxrwsr-x. 1342 lkp committer 36K 2020-12-21 19:19 c +drwxrwsr-x. 5800 lkp committer 212K 2020-12-22 03:08 p +drwxrwsr-x. 395 lkp committer 12K 2020-12-22 22:16 q +drwxrwsr-x. 1632 lkp committer 64K 2020-12-28 21:25 r +drwxrwsr-x. 445 lkp committer 20K 2020-12-28 23:16 x +drwxrwsr-x. 1796 lkp committer 56K 2021-01-05 09:20 s +drwxrwsr-x. 39 lkp committer 4.0K 2021-01-08 10:48 z +``` + +��lkp:committer��git-mirror�� +``` +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-16 10:57 git.code.sf.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-16 11:22 code.reversed.top +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-20 03:56 source.ffmpeg.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-21 02:46 git.kernel.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-22 04:05 gitlab.univ-nantes.fr +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-02 18:14 git.musl-libc.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 05:56 gitlab.dkrz.de +drwxrwsr-x. 8 lkp committer 4.0K 2020-12-05 06:17 git.sv.gnu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:17 git.ageinghacker.net +drwxrwsr-x. 11 lkp committer 4.0K 2020-12-05 06:39 git.qemu-project.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:44 git.sigxcpu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:56 git.osmocom.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 07:38 gitlab.linphone.org +drwxrwsr-x. 8 lkp committer 4.0K 2020-12-05 07:52 gitlab.gnome.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-10 14:40 webrtc.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-11 03:33 www.kismetwireless.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-11 03:34 src.fedoraproject.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-16 05:53 git.tuxfamily.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:14 chriso(a)github.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:55 nedko.arnaudov.name +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:59 git.gnome.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 18:01 android.googlesource.com +drwxrwsr-x. 5 lkp committer 4.0K 2020-12-17 18:40 gopkg.in +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 19:53 git.cryptomilk.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 20:05 git.dpkg.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 20:09 kernel.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-19 14:15 aur(a)archlinux.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-21 05:57 boringssl.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 04:44 hub.spigotmc.org +drwxrwsr-x. 14 lkp committer 4.0K 2020-12-22 04:49 repo.or.cz +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 04:52 luajit.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:02 git.assembla.com +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-22 05:07 git.purrdata.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:09 brettc(a)github.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:11 git.teknik.io +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 14:40 code.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2020-12-23 01:50 git.qemu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-23 01:51 git.merproject.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-23 01:52 code.qt.io +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-23 01:57 git.tartarus.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-27 22:17 gist.github.com +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-29 00:28 invent.kde.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-29 00:55 git.sv.nongnu.org +drwxrwsr-x. 6 lkp committer 4.0K 2020-12-29 02:57 git.savannah.gnu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-29 02:57 git.xfce.org +drwxrwsr-x. 7 lkp committer 4.0K 2020-12-29 02:58 anongit.freedesktop.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-29 02:59 git.ejiek.com +drwxrwsr-x. 32 lkp committer 4.0K 2020-12-29 03:00 gitlab.com +drwxrwsr-x. 5 lkp committer 4.0K 2020-12-30 22:00 git.torproject.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-02 05:58 skia.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-03 05:57 aur(a)aur4.archlinux.org +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-05 10:56 gitlab.freedesktop.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-05 10:59 gitlab.xiph.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-05 10:59 git.savannah.nongnu.org +drwxrwsr-x. 4 lkp committer 4.0K 2021-01-05 11:01 git.tukaani.org +drwxrwsr-x. 13 lkp committer 4.0K 2021-01-05 11:04 aur.archlinux.org +drwxrwsr-x. 6 lkp committer 4.0K 2021-01-05 11:07 bitbucket.org +drwxrwsr-x. 1664 lkp committer 40K 2021-01-07 09:08 github.com +drwxrwsr-x. 5 lkp committer 4.0K 2021-01-08 05:03 go.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-08 05:24 chromium.googlesource.com +drwxrwsr-x. 14 lkp committer 4.0K 2021-01-08 06:00 aur(a)aur.archlinux.org +``` + +��git-mirror�� +``` +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:30 coccinelle +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:45 oec-hardware +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:45 libbpf +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:46 qemu +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:51 smatch +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:51 gtk +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-21 11:02 pixz +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-31 11:26 sparse +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-31 11:27 pahole +drwxrwsr-x. 3 lkp committer 4.0K 2020-09-22 15:10 upstream-repos +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-09 19:31 linux +drwxrwsr-x. 24 lkp committer 4.0K 2020-11-10 16:40 submodules +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-02 10:00 compass-ci +``` + +��pkgbuild��make�� +``` +drwxrwsr-x. 43 lkp team 200K 2020-11-11 15:17 archlinux +drwxrwsr-x. 82 root team 4.0K 2020-12-15 18:12 pkg +drwxrwsr-x. 13 wfg team 4.0K 2020-12-16 12:55 make +``` + +��rpm��spec�� +``` +drwxrwsr-x. 3 root committer 4.0K 2020-12-08 11:15 rpmbuild-pkg +``` + +��clone�� +``` +drwxrwsr-x. 3 zhy committer 4.0K 2020-06-16 11:25 mysql-server +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:28 atomic +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:28 spinlock +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:29 mutex +drwxrwsr-x. 4 zhy committer 4.0K 2020-08-24 11:52 avx2neon +drwxrwsr-x. 5 liping committer 4.0K 2020-08-24 11:52 AvxToNeon +``` + +��compass-ci/lkp-tests/.../upstream-repos�� +``` +drwxrwsr-x. 3 wfg committer 4.0K 2020-09-14 15:26 gitee.com +``` + +�� +``` +drwxrwxrwx. 3 weijihui weijihui 4.0K 2020-07-27 15:26 intrintesttool +``` + +ansible�� +``` +drwxrwsr-x. 17 root committer 4.0K 2020-12-29 10:33 ansible-examples +``` + +�� +``` +drwxrwsr-x. 2 root committer 16K 2020-03-27 10:45 lost+found +drwxrwsr-x. 2 wfg committer 4.0K 2020-07-24 16:58 gzip +drwxrwsr-x. 5 wfg committer 4.0K 2020-08-01 11:15 cci +drwxrwsr-x. 5 wfg committer 4.0K 2020-08-04 16:27 lkp-tests +drwxrwsr-x. 4 wfg committer 4.0K 2020-09-19 14:15 crystal-ci +``` -- 2.23.0

2 1

[PATCH compass-ci] tool/es-jobs: add a new tool
by Wu Zhende 12 Jan '21

12 Jan '21

Use es-jobs to summarizes information about a group of related jobs. [Example] es-jobs submit_id=xxx -f 'id,job_state,dmesg.timestamp:last,stats.sum,stats.avg' -r '0' means: 1.query the job whose submit_id is xxx 2.I want to get id,job_state,dmesg.timestamp:last,stats.sum,stats.avg 3.refine stats.success jobs [Output] { "id": [ "crystal.597090", "crystal.597093" ], "job_state": [ "finished", "finished" ], "dmesg.timestamp:last": [ 75.256233, 48.990905 ], "stats.sum": { "kmsg.timestamp:last": 124.24713799999999, "dmesg.timestamp:last": 124.24713799999999 }, "stats.avg": { "iperf.tcp.receiver.bps": 17008962077.755077, "iperf.tcp.sender.bps": 17036843967.505566, "iperf.udp.bps": 524286.6541701214 }, "stats.count": { "stats.success": 2, "stats.success_jobs": [ "crystal.597090", "crystal.597093" ] } } Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/es-jobs | 53 +++++++++++++++++++++++++++ 2 files changed, 152 insertions(+) create mode 100644 lib/es_jobs.rb create mode 100755 sbin/es-jobs diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb new file mode 100644 index 0000000..78db8f0 --- /dev/null +++ b/lib/es_jobs.rb @@ -0,0 +1,99 @@ +# SPDX-License-Identifier: GPL-2.0-only +# frozen_string_literal: true + +LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/stats" +require_relative './es_query' + +# deal jobs search from es +class ESJobs + def initialize(es_query, my_refine = [-1], fields = []) + @es = ESQuery.new(ES_HOST, ES_PORT) + @refine = my_refine + @fields = fields + @refine_jobs = [] + @query_result = @es.multi_field_query(es_query) + @jobs = {} + @stats_count = Hash.new(0) + @stats_sum = Hash.new(0) + @stats_avg = Hash.new(0) + @stats_level = { + 0 => 'stats.success', + 1 => 'stats.unknow', + 2 => 'stats.warning', + 3 => 'stats.has_error' + } + set_defaults + deal_jobs + end + + def set_defaults + @query_result['hits']['hits'].each do |job| + @jobs[job['_id']] = job['_source'] + end + + @result = {} + @fields.each do |field| + @result[field] = [] + end + end + + def add_result_fields(job, level) + return unless @refine.include?(level) || @refine.include?(-1) + + @refine_jobs << job['id'] + @fields.each do |field| + @result[field] << job[field] if job[field] + + next unless job['stats'] + + @result[field] << job['stats'][field] if job['stats'][field] + end + end + + def deal_jobs + @jobs.each do |job_id, job| + level = deal_stats(job) + add_result_fields(job, level) + + @stats_count[@stats_level[level]] += 1 + stat_key = @stats_level[level] + '_jobs' + @stats_count[stat_key] = [] unless @stats_count.key?(stat_key) + @stats_count[stat_key] << job_id + end + end + + def deal_stats(job, level = 0) + return 1 unless job['stats'] + + job['stats'].each do |key, value| + if function_stat?(key) + @stats_sum[key] += value + else + @stats_avg[key] = (@stats_avg[key] + value) / 2 + end + + level = get_stat_level(key, level) + end + return level + end + + def get_stat_level(stat, level) + return level if level >= 3 + return 3 if stat.match(/error|fail/i) + return 2 if stat.match(/warn/i) + + return 0 + end + + def output + result = { + 'stats.count' => @stats_count + } + result['stats.sum'] = @stats_sum if @fields.include?('stats.sum') + result['stats.avg'] = @stats_avg if @fields.include?('stats.avg') + @result.merge!(result) + puts JSON.pretty_generate(@result) + end +end diff --git a/sbin/es-jobs b/sbin/es-jobs new file mode 100755 index 0000000..f075ef2 --- /dev/null +++ b/sbin/es-jobs @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require 'json' + +require_relative '../lib/es_jobs' + +def parse_argv + items = {} + ARGV.each do |item| + key, value = item.split('=') + if key && value + value_list = value.split(',') + items[key] = value_list.length > 1 ? value_list : value + end + end + items +end +opt_refine = [-1] +opt_fields = [] +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' + + opts.separator 'search_key can be id, suite, os, etc.' + opts.separator 'How to use -r' + opts.separator 'Like es-jobs submit_id=xxx -r "-1,0,1,2,3"' + opts.separator '-1 means not refine' + opts.separator '{ + -1 => null, + 0 => stats.succes, + 1 => stats.unknow, + 2 => stats.warning, + 3 => stats.has_error +}' + + opts.on('-r fields', '--refine fields', 'refine jobs') do |fields| + opt_refine = fields.split(',').map(&:to_i) + end + + opts.on('-f fields', '--fields fields', 'fields you want to see') do |fields| + opt_fields = fields.split(',') + end +end + +opt_parser.parse!(ARGV) +items = parse_argv +raise 'Please enter a query' if items.empty? + +es_jobs = ESJobs.new(items, opt_refine, opt_fields) +es_jobs.output -- 2.23.0

3 2

[PATCH v2 lkp-tests] tests/send_email: refactor send_email
by Wei Jihui 12 Jan '21

12 Jan '21

It is convenient for other tasks reuse to send email [usage] job.yaml test_task: on_fail: send_email: job_failed sleep: 1h This will send email to user when job is failed. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/email.sh | 26 ++++++++++++++++++++++++++ tests/email | 41 ----------------------------------------- tests/send_email | 21 +++++++++++++++++++++ 3 files changed, 47 insertions(+), 41 deletions(-) create mode 100755 lib/email.sh delete mode 100755 tests/email create mode 100755 tests/send_email diff --git a/lib/email.sh b/lib/email.sh new file mode 100755 index 00000000..d7c2c44a --- /dev/null +++ b/lib/email.sh @@ -0,0 +1,26 @@ +#!/bin/sh + +job_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job failed: $id + +Dear $my_name: + + Sorry to inform you that your test job is failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + +Regards +Compass-CI +" +} + +send_email() +{ + local subject=$1 + local email_content + + ${subject}_content + + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$email_content" +} diff --git a/tests/email b/tests/email deleted file mode 100755 index a5f87d42..00000000 --- a/tests/email +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/sh -# - my_email -# - info - -: ${info:=failure} - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/http.sh -. $PROJECT_DIR/lib/env.sh - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Test job failed - -Dear $my_username: - - Sorry to inform you that your test job is failed. you can click the following link to view details. - http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root - -Regards -Compass-Ci -" -} - -compose_email() -{ - [ "$info" == "failure" ] && data_failure -} - -run_email() -{ - [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" -} - -run_email diff --git a/tests/send_email b/tests/send_email new file mode 100755 index 00000000..2911d3b8 --- /dev/null +++ b/tests/send_email @@ -0,0 +1,21 @@ +#!/bin/sh +# - subject +# my_email + +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/email.sh + +[ -n "$my_email" ] || { + echo "Can not get my_email, send email failed!" + exit 0 +} + +[ -n "$subject" ] || { + echo "Can not get email subject, send email failed!" + exit 0 +} + + +send_email $subject -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..6642703 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,11 +249,30 @@ class Job set_upload_dirs() end - def get_package_dir - if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + def get_pkg_common_dir + pkg_style = nil + ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + pkg_style = @hash[item]? + break if pkg_style end + return nil unless pkg_style && pkg_style != nil + + tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] + tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] + tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + + tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + + return common_dir + end + + def get_package_dir + package_dir = "" + common_dir = get_pkg_common_dir + return package_dir unless common_dir if @hash["cci-makepkg"]? package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" @@ -265,10 +284,9 @@ class Job else package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" package_dir += ",/cci/build-config" if @hash["config"]? - else - package_dir = "" end return package_dir -- 2.23.0

1 0

[PATCH lkp-tests] tests/*: add makepkg -s operation in archlinux
by Wang Yong 12 Jan '21

12 Jan '21

use makepkg -s to auto install depends in archlinux ref: https://git.archlinux.org/pacman.git/plain/scripts/makepkg.sh.in Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 16 +++++++++++++--- tests/cci-makepkg | 14 +++++++++++--- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 31762384d..5a473a00c 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -92,7 +92,12 @@ create_cgz_name() cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" - pkg_args="-A --check --skippgpcheck" + + if [ "${os}" == "archlinux" ]; then + pkg_args="-A --check --skippgpcheck -s --needed --noconfirm" + else + pkg_args="-A --check --skippgpcheck" + fi [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}-${pkgrel}.cgz" @@ -153,8 +158,13 @@ build_source_pkg() create_cgz_name get_config - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + if [ "${os}" == "archlinux" ]; then + BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + else + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + fi local build_state="$?" diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 7d1e5cb15..e4bab41eb 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -133,10 +133,18 @@ update_softlink() build_source_pkg() { - local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" - echo "$makepkg" + if [ "${DISTRO}" == "archlinux" ]; then + local makepkg="$LKP_SRC/sbin/makepkg -A -s --needed --noconfirm --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" - PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + + else + local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" + + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + fi [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] sbin: add email-mapping
by Luan Shengde 12 Jan '21

12 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping - search all email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue email_mapping --search-all options: -n, --name name|email|tag appoint a name|email|tag to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings -s, --search search email mapping -h, --help show this message --delete-queue delete mapping queue --search-all search all email mappings the name can be either a name, an email or something like a tag. the file content consists of names and emails, like: name: email_address_d tag: email_address_d email_address_r: email_address_d can map multi name|tag|email_address to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 119 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..365aabf --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,119 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +# require_relative "#{ENV['CCI_SRC']}/lib/redis_client" +require '../lib/redis_client' + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false, + 'search_all_map' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + opts.banner += " email_mapping --search-all\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on('--search-all', 'search all email mappings') do + map_conf['search_all_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +required_opts = ['-n', '-f', '--delete-queue', '--search-all'] + +if ARGV.empty? + ARGV << '-h' +elsif (required_opts - ARGV).eql? required_opts + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = RedisClient.new('email_mapping') + if map_conf['re_mapping'] + email_mapping.reset_hash_key(r_name, d_email) + elsif map_conf['del_map'] + email_mapping.delete_hash_key(r_name) + elsif map_conf['search_map'] + mapped_email = email_mapping.search_hash_key(r_name) + if mapped_email.nil? || mapped_email.empty? + puts "#{r_name} has not been mapped an email yet." + else + puts mapped_email + end + elsif map_conf['search_all_map'] + all_mappings = email_mapping.search_all_hash_key + + all_mappings.each do |k, v| + puts "#{k}: #{v}\n" + end + elsif map_conf['delete_queue'] + email_mapping.delete_queue + else + return if email_mapping.add_hash_key(r_name, d_email) + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] lib: add redis_client
by Luan Shengde 12 Jan '21

12 Jan '21

used to: add redis hash key delete redis hash key reset redis hash key search redis hash key search all redis hash keys, values delete redis queue Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/redis_client.rb | 49 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100755 lib/redis_client.rb diff --git a/lib/redis_client.rb b/lib/redis_client.rb new file mode 100755 index 0000000..0c05252 --- /dev/null +++ b/lib/redis_client.rb @@ -0,0 +1,49 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to handle the redis queues +class RedisClient + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(queue) + @queue = queue + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_hash_key(key, value) + return false if @redis.hexists @queue, key + + @redis.hset @queue, key, value + return true + end + + def delete_hash_key(key) + @redis.hdel @queue, key + end + + def delete_queue + @redis.del @queue + end + + def reset_hash_key(key, value) + @redis.hset @queue, key, value + end + + def search_hash_key(key) + h_value = @redis.hget @queue, key + + return h_value if h_value + end + + def search_all_hash_key + h_values = @redis.hscan_each(@queue).to_h + + return h_values + end +end -- 2.23.0

1 0

[PATCH lkp-tests] setup/pgbench-setup: start postgresql
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] when after installed postgresql-server, postgres(user) is already created, so initializing the postgresql directly. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- setup/pgbench-setup | 21 +++++---------------- 1 file changed, 5 insertions(+), 16 deletions(-) diff --git a/setup/pgbench-setup b/setup/pgbench-setup index cc0ce37da..ea55b2ebc 100755 --- a/setup/pgbench-setup +++ b/setup/pgbench-setup @@ -25,28 +25,17 @@ postgres_run_log() init_postgresql() { - grep -q postgres /etc/passwd && return - - useradd --home-dir /var/lib/postgresql --no-create-home --no-user-group --gid nogroup --system postgres - data_dir=/var/lib/postgresql/data + data_dir=/var/lib/pgsql/data mkdir -p "$data_dir" - chown postgres "$data_dir" - - bin_dir=/usr/lib/postgresql/*/bin - is_clearlinux && bin_dir=/usr/bin - postgres_run $bin_dir/initdb -D "$data_dir" 2>&1 - postgresql_conf_file="$data_dir"/postgresql.conf - run_dir=/var/run/postgresql - mkdir -p "$run_dir" - chown postgres "$run_dir" - postgres_run $bin_dir/pg_ctl -D "$data_dir" start + postgresql-setup initdb + service postgresql start } -postgresql_conf_file=$(find /etc/postgresql -name postgresql.conf | head -1) - init_postgresql +postgresql_conf_file=$(find "$data_dir" -name postgresql.conf | head -1) + [ -n "$max_wal_size" ] && log_eval "echo 'max_wal_size = $max_wal_size' >> $postgresql_conf_file" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/auto_submit: only submit one branch job
by Liu Shaofei 12 Jan '21

12 Jan '21

[why] As some repositories have many branches, the git-mirror will submit jobs for each branch. So the large number of jobs are submitted for the same repository. To avoid this problem, just master branch is submitted. If repository does not have master branch, the last branch is selected to submit job. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..950d1f0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -28,16 +28,29 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) + # just submit master branch, if not master branch, submit last one branch. + def get_branch_argvs(newrefs_info) + repo = { 'branch' => '', 'commit' => '' } + newrefs_info['new_refs']['heads'].each do |branch, commit_id| - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + repo['branch'] = branch.delete_prefix('refs/heads/') + repo['commit'] = commit_id - system(real_argvs.join(' ')) + break if repo['branch'] == 'master' end + repo + end + + def submit(newrefs_info, submit_argv) + real_argvs = Array.new(submit_argv) + + repo = get_branch_argvs(newrefs_info) + + real_argvs.push("upstream_branch=#{repo['branch']}") + real_argvs.push("upstream_commit=#{repo['commit']}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{repo['commit']}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + system(real_argvs.join(' ')) end def get_argvs(newrefs_info) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 12 Jan '21

12 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci January 2021