- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests] lib/common.rb: build_pkg_error_id function add filtering rule
by Lin Jiaxin 08 Feb '21

08 Feb '21

Some software use tempfile when build. The directory is random, thus every error will be a new error_id. In fact, they are same. Now remove the random part except /tmp/lkp (makepkg argument: BUILDDIR). /tmp/gitstatus-build.BOpUcJeETC/libgit2/src/futils.c:1143:16: warning: comparison of integer expressions of different signedness: 'uint64_t' {aka 'const long unsigned int'} and 'long int' [-Wsign-compare] => libgit2/src/futils.c:warning:comparison-of-integer-expressions-of-different-signedness:'uint64_t'{aka'const-long-unsigned-int'}and'long-int'[-Wsign-compare] Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/common.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/common.rb b/lib/common.rb index cc64eb17..7deee62e 100755 --- a/lib/common.rb +++ b/lib/common.rb @@ -339,5 +339,6 @@ def build_pkg_error_id(line) line = line.tr(" ", "-") line = line.gsub(/[-_.,;:#!\[(]+$/, "") line = line.gsub(/([-_.,;:#!]){3,}/, ":") + line = line.gsub(/^\/tmp\/(?!lkp).*?\//, "") line end -- 2.23.0

1 0

[PATCH lab-z9] add host-info of b5
by Zhang Dewan 08 Feb '21

08 Feb '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/2288hv5-2s44p-384g--b5 | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 hosts/2288hv5-2s44p-384g--b5 diff --git a/hosts/2288hv5-2s44p-384g--b5 b/hosts/2288hv5-2s44p-384g--b5 new file mode 100644 index 0000000..5dbd623 --- /dev/null +++ b/hosts/2288hv5-2s44p-384g--b5 @@ -0,0 +1,14 @@ +memory: 384G +nr_ssd_partitions: 1 +ssd_partitions: + - /dev/disk/by-id/ata-MTFDDAK480TDC-1AT1ZABYY_20032653CBD6 +mac_addr: + - 6c:eb:b6:50:c0:a9 + - 6c:eb:b6:50:c0:aa + - 64:2c:ac:de:ce:51 + - 64:2c:ac:de:ce:d3 +arch: x86_64 +nr_node: 2 +nr_cpu: 88 +model_name: Intel(R) Xeon(R) Gold 6266C CPU @ 3.00GHz +ipmi_ip: 9.3.3.10 -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends/cci-makepkg: add git depend
by Wei Jihui 08 Feb '21

08 Feb '21

user maybe use git cmd when write PKGBUILD Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/cci-makepkg | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/cci-makepkg b/distro/depends/cci-makepkg index 4ed4f19e..855ac8ce 100644 --- a/distro/depends/cci-makepkg +++ b/distro/depends/cci-makepkg @@ -1,3 +1,4 @@ +git curl gnupg gettext -- 2.23.0

1 0

[PATCH lkp-tests] add tests script for openeuler os migration
by Bai Jing 08 Feb '21

08 Feb '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/openeuler_docker | 113 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100755 tests/openeuler_docker diff --git a/tests/openeuler_docker b/tests/openeuler_docker new file mode 100755 index 000000000..69d8a58a6 --- /dev/null +++ b/tests/openeuler_docker @@ -0,0 +1,113 @@ +#!/bin/sh +# - dockerfile_url + +. $LKP_SRC/lib/debug.sh + +[ -n "$dockerfile_url" ] || die "dockerfile_url is empty " + +cd $BENCHMARK_ROOT + +fit_repo() +{ + yum reinstall -y git + git clone "$dockerfile_url" || exit 0 + file_dir=$(find ./ -type f -name Dockerfile | xargs grep -E '^FROM centos:(centos|)[78]') + [ -n "$file_dir" ] || exit 0 +} + +openeuler_ready() +{ + docker load < $BENCHMARK_ROOT/openeuler_docker/openeuler-20.03-lts-v1.tar +} + +centos_docker() +{ + flag=0 + for file in $(find ./ -name Dockerfile | xargs grep -E '^FROM centos:(centos|)[78]'| awk -F: '{print $1}') + do + file=${file#*/} + image=$(echo "$file" | awk -F'/' '{print $2}' | tr 'A-Z' 'a-z') + + ( + cd $BENCHMARK_ROOT/${file%/*} + if [ -f "build.sh" ] && [ -x "build.sh" ] + then + ./build.sh > /dev/null 2> /dev/null && handle_dockerfile + else + docker_build > /dev/null 2> /dev/null && handle_dockerfile + fi + ) + done +} + +handle_FROM() +{ + sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" Dockerfile + +} + +handle_epel() +{ + grep -qw "epel-release" Dockerfile && { + sed -i "s/epel-release/bash/g" Dockerfile + } +} + +handle_epelrpm() +{ + + grep -qw "https://.*epel-release.*rpm" Dockerfile && { + sed -i "s|https://.*epel-release.*rpm|bash|g" Dockerfile + + } +} + +handle_groupadd() +{ + grep -qw "groupadd*" Dockerfile && { + sed -i '/groupadd*/i\ yum -y install shadow && \\' Dockerfile + } + +} + +handle_dockerfile() +{ + flag=1 + handle_FROM + handle_epel + handle_groupadd + docker_build "-openeuler" + [ $? -ne 0 ] || docker_run "-openeuler" +} + +docker_build() +{ + docker build -t "${image}$1" . + [ $? -ne 0 ] && { + [ "$flag" == "1" ] && echo "${image}$1.build.fail: 1" + return 1 + } + [ "$flag" == "1" ] && echo "${image}$1.build.success: 1" + flag=0 +} + +docker_run() +{ + docker run -d --name "${image}" "${image}$1" + docker_stats=$(docker inspect ${image} | grep Running | awk -F ' ' '{print $2}' | tr -d ',') + [ $docker_stats == false ] && { + echo "${image}.run.fail: 1" + return 1 + } + + echo "${image}.run.success: 1" +} + +docker_test() +{ + fit_repo + openeuler_ready + centos_docker +} + +docker_test -- 2.23.0

2 1

[PATCH compass-ci] answerback-email: add store lab to ES
by Luan Shengde 08 Feb '21

08 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 8dceee1..e2672aa 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -177,9 +177,9 @@ options.parse!(ARGV) def apply_account(my_info, conf_info) apply_info = {} + my_info['lab'] = LAB apply_info.update my_info apply_info.update conf_info - apply_info['lab'] = LAB assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" account_info_str = RestClient.post assign_account_url, apply_info.to_json -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] lib/job.cr: refactor the matching string action of os_mount
by Xu Xijian 08 Feb '21

08 Feb '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 89c92c1..b724b07 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -509,7 +509,7 @@ class Job if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) - elsif "#{os_mount}" == "nfs" || "#{os_mount}" == "cifs" || "#{os_mount}" == "local" + elsif ["nfs", "cifs", "local"].includes? "#{os_mount}" temp_initrds.concat(nfs_cifs_initrds()) end -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-find: support count keyword
by Lu Weitao 08 Feb '21

08 Feb '21

[Example] es-find tbox_group=vm2p8g -c/--count suite,job_state { "iperf": { "finished": 50, "failed": 40, "incomplete": 4, "boot": 3 }, "iperf-walk-os-test": { "finished": 167, "failed": 22 }, "borrow": { "finished": 47, "incomplete": 17, "failed": 2 }, ... } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/sbin/es-find b/sbin/es-find index ed54955..2c3b3dc 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -6,6 +6,7 @@ # Usage: # es-find key=val ... # es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." +# es-find tbox_group=vm-2p8g -c suite,job_state # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -74,6 +75,7 @@ end options = { nr_jobs: 10, desc_keyword: 'start_time' } range = nil +count = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -97,6 +99,10 @@ opt_parser = OptionParser.new do |opts| range = r end + opts.on('-c', '--count KEYWORD', 'count keyword: suite|job_state|testbox|id|...') do |c| + count = c + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -108,6 +114,11 @@ items = parse_argv items[:range] = prase_range(range) if range es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) +if count + count_keywords = count.split(',') + query_result = es.query_fields(count_keywords, items) +else + query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) +end json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

1 0

[PATCH compass-ci] lib/stats.rb: use Set save stat program
by Lu Weitao 08 Feb '21

08 Feb '21

[Why] we use some key of job.yaml as stat program, its may duplicate with default stat program for example: job.yaml include suite: openeuler_docker openeuler_docker: xxx we always insert $suite to stats_list so the 'openeuler_docker' will be extracted twice [How] use Set replace to Array Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/stats.rb b/lib/stats.rb index acd807a..c01d38d 100644 --- a/lib/stats.rb +++ b/lib/stats.rb @@ -43,7 +43,7 @@ class Stats end def assign_stats_list - stats_list = [] + stats_list = Set.new stats_list << @job['suite'] stats_list << ['time', @job['suite'] + '.time'] stats_list << 'stderr' -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/job.cr: refactor the matching string action of os_mount
by Xu Xijian 08 Feb '21

08 Feb '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 89c92c1..597c88c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -509,7 +509,7 @@ class Job if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) - elsif "#{os_mount}" == "nfs" || "#{os_mount}" == "cifs" || "#{os_mount}" == "local" + elsif ["nfs", "cifs", "local"].index "#{os_mount}" temp_initrds.concat(nfs_cifs_initrds()) end -- 2.23.0

2 2

[PATCH compass-ci] container/dracut-initrd: fix: awk: command not found
by Xu Xijian 08 Feb '21

08 Feb '21

[Why] When start an os, it will output awk: command not found in dracut stage. See errlog, this happens in 40network module, so let's install awk. [errlog] [ 7.417877] dracut-initqueue[368]: /lib/net-lib.sh: line 125: awk: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index f7119ec..8f03fa8 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -16,3 +16,4 @@ rm -rf /var/lib/apt/lists/* cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh +sed -i "/install() {/ainst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -- 2.23.0

2 2

[PATCH compass-ci] doc/manual/borrow-machine: improve readability
by Li Ping 08 Feb '21

08 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- doc/manual/borrow-machine.en.md | 18 +++++++++--------- doc/manual/borrow-machine.zh.md | 18 +++++++++--------- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index 7e66f40..72bcdcb 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -64,24 +64,24 @@ If you have not completed above works, reference to [apply-account.md](https://g # FAQ -## Customize the borrowing time +* Customize the borrowing time Find key in the yaml file and edit its value according to your requirement. - - The borrowing period can be calculated in days and hours. - - The maximum period is no more than 10 day. + The borrowing period can be calculated in days and hours. + The maximum period is no more than 10 day. -## Guidance for command submit +* Guidance for command submit See the usage and options for command 'submit' the command below: - submit -h + submit -h Reference the following line to learn the advanced usage for command 'submit': + + [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… - [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… - -## Available testbox +* Available testbox For a full list of testbox, reference to https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts @@ -96,6 +96,6 @@ If you have not completed above works, reference to [apply-account.md](https://g > - You will need to wait if there are already tasks in the task queue for the machine. > - Your job will be randomly assigned to a machine that meets the requirements if the testbox name is not end with '-axx'. -## Specify the OS +* Specify the OS About supportted `os`, `os_arch`, `os_version`, reference to [os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/ma… diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 73aea6b..c19d8a5 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -3,7 +3,7 @@ - 申请account - 配置默认yaml文件 -如果未完成以上步骤，请参考[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)完成操作。 +如果未完成以上步骤，请参考 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成操作。 # 申请测试机 @@ -62,14 +62,14 @@ # FAQ -## 自定义借用时长 +* 自定义借用时长 - 在借用机器的yaml实例文件中，找到’runtime‘字段，根据需求修改借用时长。 + 在借用机器的yaml实例文件中，找到 **runtime** 字段，根据需求修改借用时长。 - - 借用机器可以按照小时（h）、天（d）来计算。 - - 借用时长最多不超过10天。 + 借用机器可以按照小时（h）、天（d）来计算。 + 借用时长最多不超过10天。 -## submit命令指导 +* submit命令指导学习submit命令，您可以使用以下命令查看submit命令的各项参数及使用方法： @@ -77,9 +77,9 @@ 也可以参考submit命令手册学习submit命令高级用法： - [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -## 可选的testbox +* 可选的testbox 查看可选的testbox列表，请参考：https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts @@ -95,6 +95,6 @@ > - 物理机的testbox若选择以`--axx`结尾的，则表示指定到了具体的某一个物理机。若此物理机任务队列中已经有任务在排队，则需要等待队列中前面的任务执行完毕后，才会轮到你提交的borrow任务。 > - 物理机的testbox若不选择以`-axx`结尾的，表示不指定具体的某一个物理机。则此时集群中的空闲物理机会即时被分配执行你的borrow任务。 -## 如何 borrow 指定的操作系统 +* 如何 borrow 指定的操作系统关于支持的`os`, `os_arch`, `os_version`，参见：[os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/job/os-os_verison-os_arch.md) -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu/kvm.sh: delete unnecessary variable
by Liu Yinsi 08 Feb '21

08 Feb '21

before: -initrd $initrd after: -initrd concatenated-initrd [why] 1. concatenated-initrd is a file, no path involved, can use directly, so remove $initrd. 2. set_initrd() also unnecessary if directly use concatenated-initrd, so remove set_initrd(). 3. naming the file concatenated-initrd to make it more obvious and searchable. refer: [PATCH v5 compass-ci 2/2] providers/qemu: remove unnecessary variable Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 103e9e6..68b9a62 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -76,16 +76,12 @@ check_qemu() check_initrds() { - [ -n "$initrds" ] || { + if [ -n "$initrds" ]; then + cat $initrds > concatenated-initrd + else log_error "The current initrds is null." exit 1 - } -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + fi } set_bios() @@ -148,7 +144,7 @@ public_option() $qemu -name guest=$hostname,process=$job_id -kernel $kernel - -initrd $initrd + -initrd concatenated-initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -211,7 +207,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] providers/qemu: remove unnecessary variable
by Liu Yinsi 08 Feb '21

08 Feb '21

before: -initrd $initrd after: -initrd initrd [why] initrd is a file, no path involved, can use directly, so remove $initrd, set_initrd() also unnecessary if directly use initrd. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 0c58d94..1e3f397 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -69,18 +69,13 @@ check_option_value() [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 if [ -n "$initrds" ]; then - cat $initrds > $initrd + cat $initrds > initrd else log_error "The current initrds is null." exit 1 fi } -set_initrd() -{ - initrd=initrd -} - set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd @@ -137,7 +132,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -196,7 +191,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

2 2

[PATCH v3 compass-ci] container/register-accounts: fix failed to register account
by Liu Yinsi 08 Feb '21

08 Feb '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/register-accounts# ./start register-accounts register-accounts /c/compass-ci/container/register-accounts # ./register-accounts awk: /etc/compass-ci/defaults/*.yaml: No such file or directory "option: {\"my_email\"=>\"\", \"my_name\"=>\"\", \"lab\"=>\"\"}" Traceback (most recent call last): 10: from ./register-accounts:8:in `<main>' 9: from ./register-accounts:8:in `require_relative' 8: from /c/compass-ci/sbin/build-my-info.rb:57:in `<top (required)>' 7: from /c/compass-ci/sbin/build-my-info.rb:48:in `build_my_info' 6: from /c/compass-ci/lib/build_my_info_client.rb:30:in `search_my_info' 5: from /c/compass-ci/lib/es_query.rb:58:in `query_by_id' 4: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-api-7.10.1/lib/elasticsearch/api/actions/search.rb:103:in `search' 3: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/client.rb:182:in `perform_request' 2: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request' 1: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:347:in `perform_request' /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error': [400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"accounts","node":"I1usO2axQ9-XZhzcX3oHxw","reason":{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts","caused_by":{"type":"illegal_argument_exception","reason":"Ids can't be empty"}}}]},"status":400} (Elasticsearch::Transport::Transport::Errors::BadRequest) refer code: /c/compass-ci/sbin/build-my-info.rb: ... option = { 'my_email' => `git config --global user.email`.chomp, 'my_name' => `git config --global user.name`.chomp, 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } ... build_my_info(option) ... [why] in a new running container environment, not exists git config (user email, user name) and config yaml (/etc/compass-ci/defaults/*.yaml), so option is empty, then failed to register account. the purpose that require build-my-info.rb, is not execute build-my-info.rb script, we just want to call function build_my_info(). [how] when sbin/build-my-info.rb is not executed independently but invoked by other files, avoid to execute function build_my_info(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/build-my-info.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index dbe093d..098189a 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -52,4 +52,6 @@ def build_my_info(option) build_my_info.config_my_info(info_es) end -build_my_info(option) +if $PROGRAM_NAME == __FILE__ + build_my_info(option) +end -- 2.23.0

2 2

[PATCH v3 compass-ci 1/5] container/defconfig: add function docker_skip_rebuild
by Liu Yinsi 08 Feb '21

08 Feb '21

when execute `sparrow/4-docker/buildall run-only`, if caller script export env DOCKER_SKIP_REBUILD, then image already is exist, just exit build script, not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 6 ++++++ container/defconfig.sh | 8 ++++++++ 2 files changed, 14 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index f9b0824..820147a 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -47,3 +47,9 @@ def get_available_memory # take the middle value according to the system memory size. [1024, 30720, Math.sqrt(memtotal) * 1024].sort[1].to_i end + +def docker_skip_rebuild(tag) + return unless ENV.include? 'DOCKER_SKIP_REBUILD' + + exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" +end diff --git a/container/defconfig.sh b/container/defconfig.sh index 54e65b5..6f14890 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -41,3 +41,11 @@ push_image() docker tag "$src_tag" "$dst_tag" docker push "$dst_tag" } + +docker_skip_rebuild() +{ + tag=$1 + [ -z $DOCKER_SKIP_REBUILD ] && return + docker image inspect $tag > /dev/null 2>&1 + [ "$?" == "0" ] && exit 1 +} -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends/lkp: add hostname for lkp
by Wei Jihui 08 Feb '21

08 Feb '21

[error] test cluster task with initramfs, it will use hostname. /lkp/lkp/src/lib/job.sh: line 114: hostname: command not found Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/lkp b/distro/depends/lkp index 82c67184..945e29cf 100644 --- a/distro/depends/lkp +++ b/distro/depends/lkp @@ -6,3 +6,4 @@ gzip which rsync ntpdate +hostname -- 2.23.0

1 0

[PATCH compass-ci] container/send-mail: fix no package/constant
by Luan Shengde 08 Feb '21

08 Feb '21

fix no constant: from send-mail.rb:8:in `<main>' /c/compass-ci/lib/mail-post/email_init.rb:9:in `<top (required)>': uninitialized constant REDIS_HOST (NameError) fix no package: from send-mail.rb:12:in `<main>' /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- rufus-scheduler (LoadError) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-mail/Dockerfile | 2 +- container/send-mail/send-mail.rb | 4 ++++ container/send-mail/start | 1 + 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index b334b07..29f1bc6 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index 436736c..387df02 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,6 +5,10 @@ require 'mail' require 'sinatra' + +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + require "#{ENV['CCI_SRC']}/lib/mail-post" mail_server = %x(/sbin/ip route |awk '/default/ {print $3}').chomp diff --git a/container/send-mail/start b/container/send-mail/start index 95d797f..4af533d 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -11,6 +11,7 @@ docker_rm 'send-mail' names = Set.new %w[ LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/install-dependencies.sh: change install way
by Wang Yong 08 Feb '21

08 Feb '21

[before] install packages from $DISTRO detect [after] install packages from Package Manager it's not convenient when install packages in a new $DISTRO Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/install-dependencies.sh | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index b5b070f01..ddb43fbd2 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -5,28 +5,36 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -# choose install function base on DISTRIBUTION +# choose install function base on common Package Manager linux_dep() { - source /etc/os-release - case $ID in - ubuntu|debian) + get_package_manager + + case "$installer" in + apt-get) export DEBIAN_FRONTEND=noninteractive - sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + sudo "$installer" install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + ;; + dnf|yum) + sudo "$installer" install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux ;; - openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - sudo dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - else - sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - fi + pacman) + sudo "$installer" -Sy --noconfirm --needed ruby rubygems gcc make git lftp util-linux ;; *) - echo "$ID not support! please install dependencies manually." && exit 1 + echo "Unknown Package Manager! please install dependencies manually." && exit 1 ;; esac } +get_package_manager() +{ + has_cmd "yum" && installer="yum" + has_cmd "dnf" && installer="dnf" + has_cmd "apt-get" && installer="apt-get" + has_cmd "pacman" && installer="pacman" +} + mac_dep() { brew install ruby -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies.sh: change install way
by Wang Yong 08 Feb '21

08 Feb '21

[before] install packages from $DISTRO detect [after] install packages from Package Manager it's not convenient when install packages in a new $DISTRO Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index b5b070f01..ac2974ef1 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -5,28 +5,44 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -# choose install function base on DISTRIBUTION +# choose install function base on common Package Manager linux_dep() { - source /etc/os-release - case $ID in - ubuntu|debian) + get_package_manager + + case "$installer" in + apt-get) export DEBIAN_FRONTEND=noninteractive - sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + sudo "$installer" install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + ;; + dnf|yum) + sudo "$installer" install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux + ;; + pacman) + sudo "$installer" -Sy --noconfirm --needed ruby rubygems gcc make git lftp util-linux + ;; + zypper) + sudo "$installer" install -y ruby ruby-devel gcc gcc-c++ make git lftp util-linux ;; - openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - sudo dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - else - sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - fi + emerge) + sudo "$installer" ruby rubygems gcc make dev-vcs/git lftp util-linux ;; *) - echo "$ID not support! please install dependencies manually." && exit 1 + echo "Unknown Package Manager! please install dependencies manually." && exit 1 ;; esac } +get_package_manager() +{ + has_cmd "yum" && installer="yum" + has_cmd "dnf" && installer="dnf" + has_cmd "apt-get" && installer="apt-get" + has_cmd "pacman" && installer="pacman" + has_cmd "zypper" && installer="zypper" + has_cmd "emerge" && installer="emerge" +} + mac_dep() { brew install ruby -- 2.23.0

1 1

[PATCH lkp-tests] lib/common.rb: build_pkg_error_id function add filtering rule
by Lin Jiaxin 07 Feb '21

07 Feb '21

Some software use tempfile when build. The directory is random, thus every error will be a new error_id. In fact, they are same. Now remove the random part except /tmp/lkp (makepkg argument: BUILDDIR). /tmp/gitstatus-build.BOpUcJeETC/libgit2/src/futils.c:1143:16: warning: comparison of integer expressions of different signedness: 'uint64_t' {aka 'const long unsigned int'} and 'long int' [-Wsign-compare] => libgit2/src/futils.c:warning:comparison-of-integer-expressions-of-different-signedness:'uint64_t'{aka'const-long-unsigned-int'}and'long-int'[-Wsign-compare] Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/common.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/common.rb b/lib/common.rb index cc64eb17..8ba6694d 100755 --- a/lib/common.rb +++ b/lib/common.rb @@ -339,5 +339,6 @@ def build_pkg_error_id(line) line = line.tr(" ", "-") line = line.gsub(/[-_.,;:#!\[(]+$/, "") line = line.gsub(/([-_.,;:#!]){3,}/, ":") + line = line.gsub(/^\/tmp\/((?!lkp).)+?\//, "") line end -- 2.23.0

1 0

[PATCH v2 lkp-tests] makepkg: avoid some unnecessary stderr
by Lin Jiaxin 07 Feb '21

07 Feb '21

unnecessary stderr like this: ==> /tmp/stderr <== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed ^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M10 0 129 0 0 137 0 --:--:-- --:--:-- --:--:-- 137 ^M 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0^M100 255k 0 255k 0 0 110k 0 --:--:-- 0:00:02 --:--:-- 313k ==> WARNING: Skipping verification of source file PGP signatures. ==> /tmp/stderr <== fcgi-2.4.2.tar.gz ... Passed ==> /tmp/stderr <== Cloning into bare repository '//shadowsocks-libev-git'... ==> /tmp/stderr <== shadowsocks-libev-git ... Skipped shadowsocks-libev@.service ... Passed shadowsocks-libev-server@.service ... Passed shadowsocks-libev-redir@.service ... Passed shadowsocks-libev-tunnel@.service ... Passed Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- etc/makepkg.conf | 12 ++++++------ sbin/makepkg | 18 +++++++++--------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 5d73399d..b2beaedb 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -8,12 +8,12 @@ # #-- The download utilities that makepkg should use to acquire sources # Format: 'protocol::agent' -DLAGENTS=('file::/usr/bin/my_curl -gqC - -o %o %u' - 'ftp::/usr/bin/my_curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' - 'http::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'https::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'rsync::/usr/bin/rsync --no-motd -z %u %o' - 'scp::/usr/bin/scp -C %u %o') +DLAGENTS=('file::/usr/bin/my_curl -sS -gqC - -o %o %u' + 'ftp::/usr/bin/my_curl -sS -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' + 'http::/usr/bin/my_curl -sS -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'https::/usr/bin/my_curl -sS -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'rsync::/usr/bin/rsync -q --no-motd -z %u %o' + 'scp::/usr/bin/scp -q -C %u %o') # Other common tools: # /usr/bin/snarf diff --git a/sbin/makepkg b/sbin/makepkg index 498b4872..7a92ea71 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -566,7 +566,7 @@ download_git() { if [[ ! -d "$dir" ]] || dir_is_empty "$dir" ; then msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git" - if ! git clone --mirror "$url" "$dir"; then + if ! git clone -q --mirror "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -580,7 +580,7 @@ download_git() { exit 1 fi msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "git" - if ! git fetch --all -p; then + if ! git fetch -q --all -p; then # only warn on failure to allow offline builds warning "$(gettext "Failure while updating %s %s repo")" "${repo}" "git" fi @@ -609,13 +609,13 @@ extract_git() { if [[ -d "${dir##*/}" ]]; then updating=1 cd_safe "${dir##*/}" - if ! git fetch; then + if ! git fetch -q; then error "$(gettext "Failure while updating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 fi cd_safe "$srcdir" - elif ! git clone "$dir"; then + elif ! git clone -q "$dir"; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -645,7 +645,7 @@ extract_git() { fi if [[ $ref != "origin/HEAD" ]] || (( updating )) ; then - if ! git checkout --force --no-track -B makepkg $ref; then + if ! git checkout -q --force --no-track -B makepkg $ref; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -1270,10 +1270,10 @@ verify_integrity_one() { local source_name=$1 integ=$2 expectedsum=$3 local file="$(get_filename "$source_name")" - printf ' %s ... ' "$file" >&2 + printf ' %s ... ' "$file" if [[ $expectedsum = 'SKIP' ]]; then - printf '%s\n' "$(gettext "Skipped")" >&2 + printf '%s\n' "$(gettext "Skipped")" return fi @@ -1285,7 +1285,7 @@ verify_integrity_one() { local realsum="$(openssl dgst -${integ} "$file")" realsum="${realsum##* }" if [[ ${expectedsum,,} = "$realsum" ]]; then - printf '%s\n' "$(gettext "Passed")" >&2 + printf '%s\n' "$(gettext "Passed")" else printf '%s\n' "$(gettext "FAILED")" >&2 return 1 @@ -1553,7 +1553,7 @@ check_source_integrity() { warning "$(gettext "Skipping verification of source file checksums.")" check_pgpsigs "$@" elif (( SKIPPGPCHECK )); then - warning "$(gettext "Skipping verification of source file PGP signatures.")" + # warning "$(gettext "Skipping verification of source file PGP signatures.")" check_checksums "$@" else check_checksums "$@" -- 2.23.0

4 5

[PATCH v3 compass-ci] doc/write-PKGBUILD: fix documentation errors
by Lin Jiaxin 07 Feb '21

07 Feb '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- doc/manual/write-PKGBUILD.en.md | 5 +++-- doc/manual/write-PKGBUILD.zh.md | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/manual/write-PKGBUILD.en.md b/doc/manual/write-PKGBUILD.en.md index c0c25e9..c259399 100644 --- a/doc/manual/write-PKGBUILD.en.md +++ b/doc/manual/write-PKGBUILD.en.md @@ -13,8 +13,9 @@ PKGBUILD includes variables and functions. - pkgrel: Mandatory. It indicates the release number of a software package. - arch: Mandatory. It indicates the architecture sequence of a software package. - depends: Optional. It indicates the name of the dependency package required for running a software test. -- makedepends: Optional. It indicates the list of files required for building a software package. -- source: Optional. It indicates the MD5 hash value of each source file to verify the integrity of the source file during the build process. +- makedepends: Optional. It indicates the name of the dependency package required for building a software package. +- source: Optional. It indicates the list of files required for building a software package. +- md5sums: Optional. It indicates the MD5 hash value of each source file to verify the integrity of the source file during the build process. ## Defining a Function diff --git a/doc/manual/write-PKGBUILD.zh.md b/doc/manual/write-PKGBUILD.zh.md index 414bfdc..ff53f0f 100644 --- a/doc/manual/write-PKGBUILD.zh.md +++ b/doc/manual/write-PKGBUILD.zh.md @@ -13,8 +13,9 @@ PKGBUILD包含两部分内容：变量和函数。 - pkgrel: 必须定义，表示软件包的发布号； - arch: 必须定义，表示软件包使用的架构序列； - depends: 可选字段，软件测试运行时需要的依赖包名称； -- makedepends: 可选字段，构建软件包时需要的文件列表； -- source: 可选字段，指定每个源文件的MD5哈希值，用于构建过程中验证源文件的完整性。 +- makedepends: 可选字段，构建软件包时需要的依赖包名称； +- source: 可选字段，构建软件包时需要的文件列表； +- md5sums: 可选字段，指定每个源文件的MD5哈希值，用于构建过程中验证源文件的完整性。 ## 定义函数 -- 2.23.0

2 2

[PATCH v3 compass-ci 5/5] sparrow/4-docker/buildall: prevent duplicate start
by Liu Yinsi 07 Feb '21

07 Feb '21

[why] when run container which has start depends, some containers will be start many times, so avoid repeated starting containers is necessary. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index b24eca5..0d1f161 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -53,10 +53,13 @@ do_one_run() local container=$1 local container_name=$(basename $container) - cd "$container" - [ -x first-run ] && ./first-run - [ -x start ] && ./start - [ "$container_name" == 'initrd-lkp' ] && ./run + mkdir $tmpdir/start_$container_name 2>/dev/null && + ( + cd "$container" + [ -x first-run ] && ./first-run + [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run + ) } tmpdir=$(mktemp -d) -- 2.23.0

1 0

[PATCH v3 compass-ci 4/5] sparrow/4-docker/buildall: optimise function log_info() calling
by Liu Yinsi 07 Feb '21

07 Feb '21

before: just execute $container/build script, echo "build $container" after: only execute $container/build -> `docker build -t`, echo "build $container" [why] when execute `sparrow/4-docker/buildall run-only`, will execute $container/build script, but if image already exist, actually will not build container, it will exit code "1", so only exit code "0", call log_info "finish build $container" Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 83ecc9f..b24eca5 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -29,13 +29,9 @@ do_one_build() mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" - - log_info "start build $container." - [ -x build ] && ./build + [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install - - log_info "finish build $container." ) lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH v3 compass-ci 3/5] sparrow/4-docker/buildall: export env DOCKER_SKIP_REBUILD
by Liu Yinsi 07 Feb '21

07 Feb '21

if env DOCKER_SKIP_REBUILD exists, then check image exists in $container/build script by call docker_skip_rebuild(), then not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index d4e9419..83ecc9f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -70,11 +70,9 @@ do container_name=$(basename $dir) [ "$container_name" == 'ssh-r' ] && continue - docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then - build_depends $dir & - wait - fi + [ "$action" == "run-only" ] && export DOCKER_SKIP_REBUILD=DOCKER_SKIP_REBUILD + build_depends $dir & + wait start_depends $dir done -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci