February 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sbin/es-find: support count keyword
by Lu Weitao 08 Feb '21

08 Feb '21

[Example] es-find tbox_group=vm2p8g -c/--count suite,job_state { "iperf": { "finished": 50, "failed": 40, "incomplete": 4, "boot": 3 }, "iperf-walk-os-test": { "finished": 167, "failed": 22 }, "borrow": { "finished": 47, "incomplete": 17, "failed": 2 }, ... } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/sbin/es-find b/sbin/es-find index ed54955..2c3b3dc 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -6,6 +6,7 @@ # Usage: # es-find key=val ... # es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." +# es-find tbox_group=vm-2p8g -c suite,job_state # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -74,6 +75,7 @@ end options = { nr_jobs: 10, desc_keyword: 'start_time' } range = nil +count = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -97,6 +99,10 @@ opt_parser = OptionParser.new do |opts| range = r end + opts.on('-c', '--count KEYWORD', 'count keyword: suite|job_state|testbox|id|...') do |c| + count = c + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -108,6 +114,11 @@ items = parse_argv items[:range] = prase_range(range) if range es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) +if count + count_keywords = count.split(',') + query_result = es.query_fields(count_keywords, items) +else + query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) +end json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

1 0

[PATCH compass-ci] lib/stats.rb: use Set save stat program
by Lu Weitao 08 Feb '21

08 Feb '21

[Why] we use some key of job.yaml as stat program, its may duplicate with default stat program for example: job.yaml include suite: openeuler_docker openeuler_docker: xxx we always insert $suite to stats_list so the 'openeuler_docker' will be extracted twice [How] use Set replace to Array Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/stats.rb b/lib/stats.rb index acd807a..c01d38d 100644 --- a/lib/stats.rb +++ b/lib/stats.rb @@ -43,7 +43,7 @@ class Stats end def assign_stats_list - stats_list = [] + stats_list = Set.new stats_list << @job['suite'] stats_list << ['time', @job['suite'] + '.time'] stats_list << 'stderr' -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/job.cr: refactor the matching string action of os_mount
by Xu Xijian 08 Feb '21

08 Feb '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 89c92c1..597c88c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -509,7 +509,7 @@ class Job if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) - elsif "#{os_mount}" == "nfs" || "#{os_mount}" == "cifs" || "#{os_mount}" == "local" + elsif ["nfs", "cifs", "local"].index "#{os_mount}" temp_initrds.concat(nfs_cifs_initrds()) end -- 2.23.0

2 2

[PATCH compass-ci] container/dracut-initrd: fix: awk: command not found
by Xu Xijian 08 Feb '21

08 Feb '21

[Why] When start an os, it will output awk: command not found in dracut stage. See errlog, this happens in 40network module, so let's install awk. [errlog] [ 7.417877] dracut-initqueue[368]: /lib/net-lib.sh: line 125: awk: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index f7119ec..8f03fa8 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -16,3 +16,4 @@ rm -rf /var/lib/apt/lists/* cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh +sed -i "/install() {/ainst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -- 2.23.0

2 2

[PATCH compass-ci] doc/manual/borrow-machine: improve readability
by Li Ping 08 Feb '21

08 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- doc/manual/borrow-machine.en.md | 18 +++++++++--------- doc/manual/borrow-machine.zh.md | 18 +++++++++--------- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index 7e66f40..72bcdcb 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -64,24 +64,24 @@ If you have not completed above works, reference to [apply-account.md](https://g # FAQ -## Customize the borrowing time +* Customize the borrowing time Find key in the yaml file and edit its value according to your requirement. - - The borrowing period can be calculated in days and hours. - - The maximum period is no more than 10 day. + The borrowing period can be calculated in days and hours. + The maximum period is no more than 10 day. -## Guidance for command submit +* Guidance for command submit See the usage and options for command 'submit' the command below: - submit -h + submit -h Reference the following line to learn the advanced usage for command 'submit': + + [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… - [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… - -## Available testbox +* Available testbox For a full list of testbox, reference to https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts @@ -96,6 +96,6 @@ If you have not completed above works, reference to [apply-account.md](https://g > - You will need to wait if there are already tasks in the task queue for the machine. > - Your job will be randomly assigned to a machine that meets the requirements if the testbox name is not end with '-axx'. -## Specify the OS +* Specify the OS About supportted `os`, `os_arch`, `os_version`, reference to [os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/ma… diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 73aea6b..c19d8a5 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -3,7 +3,7 @@ - 申请account - 配置默认yaml文件 -如果未完成以上步骤，请参考[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)完成操作。 +如果未完成以上步骤，请参考 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成操作。 # 申请测试机 @@ -62,14 +62,14 @@ # FAQ -## 自定义借用时长 +* 自定义借用时长 - 在借用机器的yaml实例文件中，找到’runtime‘字段，根据需求修改借用时长。 + 在借用机器的yaml实例文件中，找到 **runtime** 字段，根据需求修改借用时长。 - - 借用机器可以按照小时（h）、天（d）来计算。 - - 借用时长最多不超过10天。 + 借用机器可以按照小时（h）、天（d）来计算。 + 借用时长最多不超过10天。 -## submit命令指导 +* submit命令指导学习submit命令，您可以使用以下命令查看submit命令的各项参数及使用方法： @@ -77,9 +77,9 @@ 也可以参考submit命令手册学习submit命令高级用法： - [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -## 可选的testbox +* 可选的testbox 查看可选的testbox列表，请参考：https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts @@ -95,6 +95,6 @@ > - 物理机的testbox若选择以`--axx`结尾的，则表示指定到了具体的某一个物理机。若此物理机任务队列中已经有任务在排队，则需要等待队列中前面的任务执行完毕后，才会轮到你提交的borrow任务。 > - 物理机的testbox若不选择以`-axx`结尾的，表示不指定具体的某一个物理机。则此时集群中的空闲物理机会即时被分配执行你的borrow任务。 -## 如何 borrow 指定的操作系统 +* 如何 borrow 指定的操作系统关于支持的`os`, `os_arch`, `os_version`，参见：[os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/job/os-os_verison-os_arch.md) -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu/kvm.sh: delete unnecessary variable
by Liu Yinsi 08 Feb '21

08 Feb '21

before: -initrd $initrd after: -initrd concatenated-initrd [why] 1. concatenated-initrd is a file, no path involved, can use directly, so remove $initrd. 2. set_initrd() also unnecessary if directly use concatenated-initrd, so remove set_initrd(). 3. naming the file concatenated-initrd to make it more obvious and searchable. refer: [PATCH v5 compass-ci 2/2] providers/qemu: remove unnecessary variable Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 103e9e6..68b9a62 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -76,16 +76,12 @@ check_qemu() check_initrds() { - [ -n "$initrds" ] || { + if [ -n "$initrds" ]; then + cat $initrds > concatenated-initrd + else log_error "The current initrds is null." exit 1 - } -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + fi } set_bios() @@ -148,7 +144,7 @@ public_option() $qemu -name guest=$hostname,process=$job_id -kernel $kernel - -initrd $initrd + -initrd concatenated-initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -211,7 +207,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] providers/qemu: remove unnecessary variable
by Liu Yinsi 08 Feb '21

08 Feb '21

before: -initrd $initrd after: -initrd initrd [why] initrd is a file, no path involved, can use directly, so remove $initrd, set_initrd() also unnecessary if directly use initrd. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 0c58d94..1e3f397 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -69,18 +69,13 @@ check_option_value() [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 if [ -n "$initrds" ]; then - cat $initrds > $initrd + cat $initrds > initrd else log_error "The current initrds is null." exit 1 fi } -set_initrd() -{ - initrd=initrd -} - set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd @@ -137,7 +132,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -196,7 +191,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

2 2

[PATCH v3 compass-ci] container/register-accounts: fix failed to register account
by Liu Yinsi 08 Feb '21

08 Feb '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/register-accounts# ./start register-accounts register-accounts /c/compass-ci/container/register-accounts # ./register-accounts awk: /etc/compass-ci/defaults/*.yaml: No such file or directory "option: {\"my_email\"=>\"\", \"my_name\"=>\"\", \"lab\"=>\"\"}" Traceback (most recent call last): 10: from ./register-accounts:8:in `<main>' 9: from ./register-accounts:8:in `require_relative' 8: from /c/compass-ci/sbin/build-my-info.rb:57:in `<top (required)>' 7: from /c/compass-ci/sbin/build-my-info.rb:48:in `build_my_info' 6: from /c/compass-ci/lib/build_my_info_client.rb:30:in `search_my_info' 5: from /c/compass-ci/lib/es_query.rb:58:in `query_by_id' 4: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-api-7.10.1/lib/elasticsearch/api/actions/search.rb:103:in `search' 3: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/client.rb:182:in `perform_request' 2: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request' 1: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:347:in `perform_request' /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error': [400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"accounts","node":"I1usO2axQ9-XZhzcX3oHxw","reason":{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts","caused_by":{"type":"illegal_argument_exception","reason":"Ids can't be empty"}}}]},"status":400} (Elasticsearch::Transport::Transport::Errors::BadRequest) refer code: /c/compass-ci/sbin/build-my-info.rb: ... option = { 'my_email' => `git config --global user.email`.chomp, 'my_name' => `git config --global user.name`.chomp, 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } ... build_my_info(option) ... [why] in a new running container environment, not exists git config (user email, user name) and config yaml (/etc/compass-ci/defaults/*.yaml), so option is empty, then failed to register account. the purpose that require build-my-info.rb, is not execute build-my-info.rb script, we just want to call function build_my_info(). [how] when sbin/build-my-info.rb is not executed independently but invoked by other files, avoid to execute function build_my_info(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/build-my-info.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index dbe093d..098189a 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -52,4 +52,6 @@ def build_my_info(option) build_my_info.config_my_info(info_es) end -build_my_info(option) +if $PROGRAM_NAME == __FILE__ + build_my_info(option) +end -- 2.23.0

2 2

[PATCH v3 compass-ci 1/5] container/defconfig: add function docker_skip_rebuild
by Liu Yinsi 08 Feb '21

08 Feb '21

when execute `sparrow/4-docker/buildall run-only`, if caller script export env DOCKER_SKIP_REBUILD, then image already is exist, just exit build script, not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 6 ++++++ container/defconfig.sh | 8 ++++++++ 2 files changed, 14 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index f9b0824..820147a 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -47,3 +47,9 @@ def get_available_memory # take the middle value according to the system memory size. [1024, 30720, Math.sqrt(memtotal) * 1024].sort[1].to_i end + +def docker_skip_rebuild(tag) + return unless ENV.include? 'DOCKER_SKIP_REBUILD' + + exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" +end diff --git a/container/defconfig.sh b/container/defconfig.sh index 54e65b5..6f14890 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -41,3 +41,11 @@ push_image() docker tag "$src_tag" "$dst_tag" docker push "$dst_tag" } + +docker_skip_rebuild() +{ + tag=$1 + [ -z $DOCKER_SKIP_REBUILD ] && return + docker image inspect $tag > /dev/null 2>&1 + [ "$?" == "0" ] && exit 1 +} -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends/lkp: add hostname for lkp
by Wei Jihui 08 Feb '21

08 Feb '21

[error] test cluster task with initramfs, it will use hostname. /lkp/lkp/src/lib/job.sh: line 114: hostname: command not found Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/lkp b/distro/depends/lkp index 82c67184..945e29cf 100644 --- a/distro/depends/lkp +++ b/distro/depends/lkp @@ -6,3 +6,4 @@ gzip which rsync ntpdate +hostname -- 2.23.0

1 0