- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] delimiter: use assist-result container client to get job yaml
by Cao Xueliang 14 Jan '21

14 Jan '21

Before: Get the job yaml from es, we need deal some process products of job. After: Get the job yaml from assist-result container without the process products. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 2 +- src/delimiter/utils.rb | 24 +++++++----------------- 2 files changed, 8 insertions(+), 18 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index ebf90cc..7b14558 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,7 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' +DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index de5a302..61fe6ab 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -9,6 +9,7 @@ require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/assist_result_client' require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" @@ -121,9 +122,10 @@ module Utils end def init_job_content(job_id) - es = ESQuery.new - job = es.query_by_id(job_id) - raise "es query job id: #{job_id} failed!" unless job + job_yaml = AssistResult.new.get_job_yaml(job_id) + raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + + job = JSON.parse job_yaml account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info @@ -135,21 +137,9 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id - job['testbox'] = job['config'] ? 'dc-16g' : 'dc-8g' - job['os'] = 'openeuler' - job['os_arch'] = 'aarch64' - job['os_mount'] = 'container' - job['os_version'] = '20.03-pre' - job['docker_image'] = 'openeuler:20.03-pre' - - job.delete('start_time') - job.delete('tbox_group') - job.delete('error_ids') - job.delete('job_state') - job.delete('end_time') - job.delete('loadavg') + + job.delete('subqueue') job.delete('queue') - job.delete('stats') job.delete('id') return job -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add assist_result_client for assist_result container
by Cao Xueliang 14 Jan '21

14 Jan '21

[why] Add client for assist_result container, others can call the assist_result container API by the client. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/assist_result_client.rb | 21 +++++++++++++++++++++ lib/constants.rb | 3 +++ 2 files changed, 24 insertions(+) create mode 100644 lib/assist_result_client.rb diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb new file mode 100644 index 0000000..980486d --- /dev/null +++ b/lib/assist_result_client.rb @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2021 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true +require 'rest-client' +require_relative 'constants' + +class AssistResult + def initialize(host = ASSIST_RESULT_HOST, port = ASSIST_RESULT_PORT) + @host = host + @port = port + end + + def get_job_yaml(job_id) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/get_job_yaml/#{job_id}") + response = resource.get() + return nil unless response.code == 200 + + return response.body + end +end diff --git a/lib/constants.rb b/lib/constants.rb index a274adc..ffc4f03 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -15,3 +15,6 @@ SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' + +ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' +ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 -- 2.23.0

1 0

Re: [PATCH v2 lkp-tests 2/2] lib: fix 409 conflict
by Wu Fengguang 14 Jan '21

14 Jan '21

On Wed, Jan 13, 2021 at 09:19:47PM +0800, Li Ping wrote: >[why] >find a -type d >a >a/b >a/c >find a -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >==> curl: (22) The requested URL returned error: 409 Conflict >reason: upload an empty folder causes this error. I suspect 409 happens because server side already have that dir. It's like you run mkdir a => success mkdir a => 409 But will the server side auto create new subdirs if they do not already exist? Probably the below find will fail. Please experiment with deeper dirs, like a/b/c/file Please be very careful and make sure you understand all the error messages, all situations and the root cause before going ahead changing something. >[how] >find a -type f -exec curl -sSf -X MKCOL "$http_url/{}" \; >a/b >a/c >All the files would be found, there is no need to upload empty folder. Thanks, Fengguang

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Bai Jing 14 Jan '21

14 Jan '21

On Wed, Jan 13, 2021 at 09:19:46PM +0800, Li Ping wrote: >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > distro/depends/api-avx2neon | 4 ++++ > tests/api-avx2neon | 33 ++++++++++++++++++++++----------- > 2 files changed, 26 insertions(+), 11 deletions(-) > create mode 100644 distro/depends/api-avx2neon > >diff --git a/distro/depends/api-avx2neon b/distro/depends/api-avx2neon >new file mode 100644 >index 000000000..d82d2f65e >--- /dev/null >+++ b/distro/depends/api-avx2neon >@@ -0,0 +1,4 @@ >+git >+make >+gcc >+g++ >diff --git a/tests/api-avx2neon b/tests/api-avx2neon >index 9a64f89bf..47421021c 100755 >--- a/tests/api-avx2neon >+++ b/tests/api-avx2neon >@@ -1,18 +1,29 @@ > #!/bin/bash > >+SCRIPT_DIR=$(dirname $(realpath $0)) >+cd "$SCRIPT_DIR" >+: "${git_url:=https://github.com/Siguyi/AvxToNeon}" > >-script=$(dirname $(realpath $0)) >-cd $script >+build() { >+ git clone "$git_url" >+ cd AvxToNeon/tests || exit > >-yum install -y git >-yum install -y make >+ if [ "${arch}" == "aarch64" ]; then >+ make >/dev/null 2>&1 >+ elif [ "${arch}" == "x86_64" ]; then >+ make ARCH=x86 >/dev/null 2>&1 >+ fi >+} > >-git clone https://github.com/Siguyi/AvxToNeon >+run() { >+ build >+ echo "api-avx2neon running" >+ ./test >+ echo "api-avx2neon finished" > >-cd AvxToNeon >-cd tests >-make >+ output=$(find / -name "output") >+ fail=$(grep 'AVX2NEONTest Complete' $output |awk '{print $8}') ^^ need space Thanks, Baijing >+ [ 0 -eq "$fail" ] || exit 1 >+} > >-echo "api-avx2neon running" >-./test >-echo "api-avx2neon finished" >+run >-- >2.23.0 >

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Wang Yong 14 Jan '21

14 Jan '21

On Wed, Jan 13, 2021 at 09:19:46PM +0800, Li Ping wrote: >Signed-off-by: Li Ping <1477412247(a)qq.com> >+ if [ "${arch}" == "aarch64" ]; then >+ make >/dev/null 2>&1 >+ elif [ "${arch}" == "x86_64" ]; then >+ make ARCH=x86 >/dev/null 2>&1 >+ fi avx2neon use cross compile to build x86 version, not only require arch=x86_64 ref: Makefile Thanks, Yong

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Wu Fengguang 14 Jan '21

14 Jan '21

On Wed, Jan 13, 2021 at 09:19:46PM +0800, Li Ping wrote: >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > distro/depends/api-avx2neon | 4 ++++ > tests/api-avx2neon | 33 ++++++++++++++++++++++----------- > 2 files changed, 26 insertions(+), 11 deletions(-) > create mode 100644 distro/depends/api-avx2neon > >diff --git a/distro/depends/api-avx2neon b/distro/depends/api-avx2neon >new file mode 100644 >index 000000000..d82d2f65e >--- /dev/null >+++ b/distro/depends/api-avx2neon >@@ -0,0 +1,4 @@ >+git >+make >+gcc >+g++ >diff --git a/tests/api-avx2neon b/tests/api-avx2neon >index 9a64f89bf..47421021c 100755 >--- a/tests/api-avx2neon >+++ b/tests/api-avx2neon >@@ -1,18 +1,29 @@ > #!/bin/bash > >+SCRIPT_DIR=$(dirname $(realpath $0)) >+cd "$SCRIPT_DIR" >+: "${git_url:=https://github.com/Siguyi/AvxToNeon}" > >-script=$(dirname $(realpath $0)) >-cd $script >+build() { >+ git clone "$git_url" Better to add "|| exit" here too. >+ cd AvxToNeon/tests || exit > >-yum install -y git >-yum install -y make >+ if [ "${arch}" == "aarch64" ]; then >+ make >/dev/null 2>&1 Why quiet all output and stderr? >+ elif [ "${arch}" == "x86_64" ]; then >+ make ARCH=x86 >/dev/null 2>&1 >+ fi >+} Thanks, Fengguang >-git clone https://github.com/Siguyi/AvxToNeon >+run() { >+ build >+ echo "api-avx2neon running" >+ ./test What if ./test exit 1? >+ echo "api-avx2neon finished" > >-cd AvxToNeon >-cd tests >-make >+ output=$(find / -name "output") output= => output_file= What if find returns nothing? >+ fail=$(grep 'AVX2NEONTest Complete' $output |awk '{print $8}') Please give example line that you want to grep/awk. >+ [ 0 -eq "$fail" ] || exit 1 That logic reads awkward. fail => complete? >+} > >-echo "api-avx2neon running" >-./test >-echo "api-avx2neon finished" >+run >-- >2.23.0 >

1 0

[PATCH v3 lkp-tests] tests/*: add makepkg -s operation in archlinux
by Wang Yong 13 Jan '21

13 Jan '21

use makepkg -s to auto install depends in archlinux ref: https://git.archlinux.org/pacman.git/plain/scripts/makepkg.sh.in Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 7 ++++++- tests/cci-makepkg | 12 +++++++++--- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 31762384d..554cd56e5 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -92,7 +92,9 @@ create_cgz_name() cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" + pkg_args="-A --check --skippgpcheck" + [ "${os}" == "archlinux" ] && pkg_args="${pkg_args} -s --needed --noconfirm" [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}-${pkgrel}.cgz" @@ -153,7 +155,10 @@ build_source_pkg() create_cgz_name get_config - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + local mark="true" + [ "${os}" == "archlinux" ] && local mark="pacman" + + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf local build_state="$?" diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 7d1e5cb15..ecbf72134 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -133,10 +133,16 @@ update_softlink() build_source_pkg() { - local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" - echo "$makepkg" + local mark="true" + local pkg_args="-A --check --skippgpcheck" + [ "${DISTRO}" == "archlinux" ] && { + local mark="pacman" + local pkg_args="${pkg_args} -s --needed --noconfirm" + } + local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" + echo "${makepkg}" - PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0

[PATCH compass-ci 3/3] providers/qemu: check kernel
by Liu Yinsi 13 Jan '21

13 Jan '21

test: /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-x86_64/z9.351229 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-riscv64/z9.351202 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-aarch64/z9.351187 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 8979da6..51357a6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -57,6 +57,9 @@ check_option_value() exit } + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 + [ -n "$initrds" ] || exit } @@ -104,9 +107,6 @@ set_qemu() do command -v "$qemu" > /dev/null && break done - - # debian has both qemu-system-x86_64 and qemu-system-riscv64 command - [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 } print_message() @@ -195,9 +195,9 @@ check_logfile write_logfile parse_ipxe_script -check_option_value set_options +check_option_value print_message -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] providers/qemu: check kernel
by Liu Yinsi 13 Jan '21

13 Jan '21

[why] check $kernel includes "riscv64" to set qemu command, this logic is suitable in check_option_value(), not in set_qemu, so move order. test: /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-x86_64/z9.351229 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-riscv64/z9.351202 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-aarch64/z9.351187 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 8979da6..51357a6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -57,6 +57,9 @@ check_option_value() exit } + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 + [ -n "$initrds" ] || exit } @@ -104,9 +107,6 @@ set_qemu() do command -v "$qemu" > /dev/null && break done - - # debian has both qemu-system-x86_64 and qemu-system-riscv64 command - [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 } print_message() @@ -195,9 +195,9 @@ check_logfile write_logfile parse_ipxe_script -check_option_value set_options +check_option_value print_message -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] providers/qemu: check qemu command exist
by Liu Yinsi 13 Jan '21

13 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 9d15f7d..8979da6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -102,7 +102,7 @@ set_qemu() for qemu in "${qemus[@]}" do - [ -n "$(command -v ${qemu})" ] && break + command -v "$qemu" > /dev/null && break done # debian has both qemu-system-x86_64 and qemu-system-riscv64 command -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci