- Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci 4/4] docker-rootfs: list of packages that need to install
by Wang Chenglong 10 Dec '20

10 Dec '20

write the software in this file, it will install into rootfs. the kernel and kernel-devel is necessary. They provide the default kernel as well as kernel source code. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/packages-to-install | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 container/docker-rootfs/packages-to-install diff --git a/container/docker-rootfs/packages-to-install b/container/docker-rootfs/packages-to-install new file mode 100644 index 0000000..d7e890f --- /dev/null +++ b/container/docker-rootfs/packages-to-install @@ -0,0 +1,3 @@ +kernel +kernel-devel +openssh-server -- 2.23.0

1 0

[PATCH v6 compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 10 Dec '20

10 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..8e2e88a --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH v6 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 10 Dec '20

10 Dec '20

[Why] generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install [How] Usage: ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: ./run /tmp/openEuler-docker/openEuler-docker..tar.xz /tmp/openeuler-rootfs/ Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..cf91337 --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_aarch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

1 0

[PATCH compass-ci] keep curl command silent mode and show error
by Liu Yinsi 09 Dec '20

09 Dec '20

use '-sS' to avoid unnecessary output when add es index. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/start | 2 +- sbin/es-accounts-mapping.sh | 4 ++-- sbin/es-jobs-mapping.sh | 6 +++--- sbin/es-regression-mapping.sh | 4 ++-- sbin/es-repo-mapping.sh | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/container/logging-es/start b/container/logging-es/start index 25ed36b..66464e6 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -23,7 +23,7 @@ cmd=( echo "Please wait 60s for setting index" sleep 60 -curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' { "index": { "refresh_interval": "120s", diff --git a/sbin/es-accounts-mapping.sh b/sbin/es-accounts-mapping.sh index 635f4fc..d5566dc 100755 --- a/sbin/es-accounts-mapping.sh +++ b/sbin/es-accounts-mapping.sh @@ -3,14 +3,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # check whether accounts index has created -status_code=$(curl -sIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/accounts) +status_code=$(curl -sSIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/accounts) if [ "$status_code" -eq 200 ] then echo "accounts index has been created, exit." else echo "begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -d '{ "mappings": { "_doc": { "dynamic": false, diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 8e19662..81f19aa 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -28,14 +28,14 @@ then fi # Determine whether jobs index has created -status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/jobs) +status_code=$(curl -sSIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/jobs) if [ $status_code -eq 200 ] then echo "jobs index has create, exit." else echo "jobs index not exists, begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ "mappings": { "_doc": { "dynamic": false, @@ -151,7 +151,7 @@ else echo "create jobs index failed." else echo "set index.mapping.total_fields.limit: 10000" - curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ + curl -sS -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ -d '{"index.mapping.total_fields.limit": 10000}' fi fi diff --git a/sbin/es-regression-mapping.sh b/sbin/es-regression-mapping.sh index 7ef19cd..8e04bdb 100755 --- a/sbin/es-regression-mapping.sh +++ b/sbin/es-regression-mapping.sh @@ -1,14 +1,14 @@ #!/bin/sh # check whether regression index has created -status_code=$(curl -sIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/regression) +status_code=$(curl -sSIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/regression) if [ "$status_code" -eq 200 ] then echo "regression index has been created, exit." else echo "begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -d '{ "mappings": { "_doc": { "dynamic": false, diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh index 4605051..17f0986 100755 --- a/sbin/es-repo-mapping.sh +++ b/sbin/es-repo-mapping.sh @@ -3,12 +3,12 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # Check if "repo" index already exists. -status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +status_code=$(curl -sSIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) [ $status_code -eq 200 ] && echo '"repo" index already exists.' && exit # Create "repo" index. echo 'Start to create "repo" index.' -curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' { "mappings": { "_doc": { -- 2.23.0

2 4

[PATCH v3 lkp-tests] automatic compilation and configure the variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- install.sh | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 install.sh diff --git a/install.sh b/install.sh new file mode 100644 index 00000000..4bcc1afb --- /dev/null +++ b/install.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +. lib/env.sh + +make + +source $(shell_profile) -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add SCHED_HOST,SCHED_PORT description
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- doc/job/SCHED_HOST-SCHED_PORT.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/job/SCHED_HOST-SCHED_PORT.md diff --git a/doc/job/SCHED_HOST-SCHED_PORT.md b/doc/job/SCHED_HOST-SCHED_PORT.md new file mode 100644 index 0000000..83d86d7 --- /dev/null +++ b/doc/job/SCHED_HOST-SCHED_PORT.md @@ -0,0 +1,8 @@ +# SCHED_HOST, SCHED_PORT +Meaning: +- The job task depends on the scheduler, that will place the our task on the +test machine for running, so the scheduler needs two variables. +- `Scheduler` is one of our service, `SCHED` is the abbreviation of `Scheduler` + these variables mean: + SCHED_HOST: the scheduler service ip. + SCHED_PORT: the scheduler service port. -- 2.23.0

2 1

[PATCH lkp-tests] spec/submit_spec.rb: change spec describe
by Wei Jihui 09 Dec '20

09 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 77b64a7a..827c926e 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -35,7 +35,7 @@ def submit_job() end describe 'submit job spec' do - it 'link jobs spec' do + it 'spec for submit/*/job.yaml' do submit_job() end end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix incorrect URL address
by Liu Shaofei 09 Dec '20

09 Dec '20

==> Retrieving sources... ==> ERROR: //172.168.131.113 is not a clone of git://172.168.131.113/ Aborting... Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 17157771..8a138055 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -106,7 +106,9 @@ replace_source() for url in ${source[@]} do echo "$url" | egrep 'git\+|\.git$' && { - url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + [ -n "$repo_dir" ] && { + url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + } } url=$(echo "$url" | sed 's|https://|http://|g') -- 2.23.0

1 0

[PATCH v2 lkp-tests] lkp-tests/tests: add rpmbuild-pkg to build RPM
by Zhang Yale 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/rpmbuild-pkg | 66 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100755 tests/rpmbuild-pkg diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg new file mode 100755 index 00000000..d70b7c94 --- /dev/null +++ b/tests/rpmbuild-pkg @@ -0,0 +1,66 @@ +#!/bin/bash +# - rpm_repo + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh + +check_vars() +{ + [ -n "$rpm_repo" ] || die "rpm_repo is empty" +} + +benchmark=${rpm_repo##*/} + +RPM_MNT=/initrd/rpmbuild-pkg +mkdir -p "$RPM_MNT" +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} +sync_dest="$RPM_MNT/$pack_to" + +rpmdev-setuptree # generate workspace in $HOME + +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" +} + +request_pkg() +{ + filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + + local dir="SOURCES" + for pkgfile in ${filelist[*]} + do + echo $pkgfile | egrep "\.spec$" && { + dir="SPECS" + } + get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + done +} + +rpmbuild_pkg() +{ + spec_dir=$HOME/rpmbuild/SPECS/$benchmark.spec + #sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $benchmark.spec` + # Install build depends + yum-builddep -y $spec_dir + # Download tar.gz to default path $HOME/rpmbuild/SOURCE + spectool -g -R $spec_dir + # Building rpm or srpm packages + rpmbuild -ba $spec_dir +} + +upload_pkg() +{ + #rpm package will be generated in "$HOME/rpmbuild/SRPMS" and "$HOME/rpmbuild/RPMS" + for file in $(find $HOME/rpmbuild/ -type f -name "*.rpm") + do + upload_one_curl ${file} ${sync_dest} + done +} + +check_vars +request_pkg +rpmbuild_pkg +upload_pkg -- 2.23.0

3 4

[PATCH compass-ci] delimiter: use lkp-tests to upload bisect.log
by Cao Xueliang 09 Dec '20

09 Dec '20

We create a file and put it in "/tmp/lkp/result" dir, the lkp-tests will upload the file when finished the job. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 1 + src/delimiter/constants.rb | 2 ++ src/delimiter/utils.rb | 10 ++-------- 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 7543646..b54be4d 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -81,6 +81,7 @@ class GitBisect if temp[0].include? 'Bisecting' result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} "#{@error_id}" #{@work_dir}` end + Utils.create_bisect_log(@work_dir) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) puts "\nbisect result: #{result}" analyse_result(result) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 5e5406f..7097c6f 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -11,3 +11,5 @@ TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" RESULT_WEBDAV_URL = ENV['RESULT_WEBDAV_URL'] || "http://172.17.0.1:3080" +# The files which are in this dir can be uploaded by lkp-tests +TMP_RESULT_ROOT = '/tmp/lkp/result' diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 60b64a1..b60ada0 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -122,18 +122,12 @@ module Utils return bisect_log_arr end - def create_bisect_log(job_id, git_dir) - log_file = File.join(git_dir, "#{job_id}_bisect.log") + def create_bisect_log(git_dir) + log_file = File.join(TMP_RESULT_ROOT, "bisect.log") log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| log_content.each { |line| f.puts(line) } end - - return log_file - end - - def upload_bisect_log(log_file, dest_dir, access_key) - %x(curl -sSf -T "#{log_file}" #{RESULT_WEBDAV_URL}/#{dest_dir} --cookie "ACCESSKEY=#{access_key}") end end end -- 2.23.0

1 1