- Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 10 Dec '20

10 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..8e2e88a --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH v6 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 10 Dec '20

10 Dec '20

[Why] generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install [How] Usage: ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: ./run /tmp/openEuler-docker/openEuler-docker..tar.xz /tmp/openeuler-rootfs/ Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..cf91337 --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_aarch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

1 0

[PATCH compass-ci] keep curl command silent mode and show error
by Liu Yinsi 09 Dec '20

09 Dec '20

use '-sS' to avoid unnecessary output when add es index. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/start | 2 +- sbin/es-accounts-mapping.sh | 4 ++-- sbin/es-jobs-mapping.sh | 6 +++--- sbin/es-regression-mapping.sh | 4 ++-- sbin/es-repo-mapping.sh | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/container/logging-es/start b/container/logging-es/start index 25ed36b..66464e6 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -23,7 +23,7 @@ cmd=( echo "Please wait 60s for setting index" sleep 60 -curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' { "index": { "refresh_interval": "120s", diff --git a/sbin/es-accounts-mapping.sh b/sbin/es-accounts-mapping.sh index 635f4fc..d5566dc 100755 --- a/sbin/es-accounts-mapping.sh +++ b/sbin/es-accounts-mapping.sh @@ -3,14 +3,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # check whether accounts index has created -status_code=$(curl -sIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/accounts) +status_code=$(curl -sSIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/accounts) if [ "$status_code" -eq 200 ] then echo "accounts index has been created, exit." else echo "begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -d '{ "mappings": { "_doc": { "dynamic": false, diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 8e19662..81f19aa 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -28,14 +28,14 @@ then fi # Determine whether jobs index has created -status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/jobs) +status_code=$(curl -sSIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/jobs) if [ $status_code -eq 200 ] then echo "jobs index has create, exit." else echo "jobs index not exists, begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ "mappings": { "_doc": { "dynamic": false, @@ -151,7 +151,7 @@ else echo "create jobs index failed." else echo "set index.mapping.total_fields.limit: 10000" - curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ + curl -sS -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ -d '{"index.mapping.total_fields.limit": 10000}' fi fi diff --git a/sbin/es-regression-mapping.sh b/sbin/es-regression-mapping.sh index 7ef19cd..8e04bdb 100755 --- a/sbin/es-regression-mapping.sh +++ b/sbin/es-regression-mapping.sh @@ -1,14 +1,14 @@ #!/bin/sh # check whether regression index has created -status_code=$(curl -sIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/regression) +status_code=$(curl -sSIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/regression) if [ "$status_code" -eq 200 ] then echo "regression index has been created, exit." else echo "begin create index." - curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -d '{ "mappings": { "_doc": { "dynamic": false, diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh index 4605051..17f0986 100755 --- a/sbin/es-repo-mapping.sh +++ b/sbin/es-repo-mapping.sh @@ -3,12 +3,12 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # Check if "repo" index already exists. -status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +status_code=$(curl -sSIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) [ $status_code -eq 200 ] && echo '"repo" index already exists.' && exit # Create "repo" index. echo 'Start to create "repo" index.' -curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' { "mappings": { "_doc": { -- 2.23.0

2 4

[PATCH v3 lkp-tests] automatic compilation and configure the variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- install.sh | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 install.sh diff --git a/install.sh b/install.sh new file mode 100644 index 00000000..4bcc1afb --- /dev/null +++ b/install.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +. lib/env.sh + +make + +source $(shell_profile) -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add SCHED_HOST,SCHED_PORT description
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- doc/job/SCHED_HOST-SCHED_PORT.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/job/SCHED_HOST-SCHED_PORT.md diff --git a/doc/job/SCHED_HOST-SCHED_PORT.md b/doc/job/SCHED_HOST-SCHED_PORT.md new file mode 100644 index 0000000..83d86d7 --- /dev/null +++ b/doc/job/SCHED_HOST-SCHED_PORT.md @@ -0,0 +1,8 @@ +# SCHED_HOST, SCHED_PORT +Meaning: +- The job task depends on the scheduler, that will place the our task on the +test machine for running, so the scheduler needs two variables. +- `Scheduler` is one of our service, `SCHED` is the abbreviation of `Scheduler` + these variables mean: + SCHED_HOST: the scheduler service ip. + SCHED_PORT: the scheduler service port. -- 2.23.0

2 1

[PATCH lkp-tests] spec/submit_spec.rb: change spec describe
by Wei Jihui 09 Dec '20

09 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 77b64a7a..827c926e 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -35,7 +35,7 @@ def submit_job() end describe 'submit job spec' do - it 'link jobs spec' do + it 'spec for submit/*/job.yaml' do submit_job() end end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix incorrect URL address
by Liu Shaofei 09 Dec '20

09 Dec '20

==> Retrieving sources... ==> ERROR: //172.168.131.113 is not a clone of git://172.168.131.113/ Aborting... Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 17157771..8a138055 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -106,7 +106,9 @@ replace_source() for url in ${source[@]} do echo "$url" | egrep 'git\+|\.git$' && { - url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + [ -n "$repo_dir" ] && { + url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + } } url=$(echo "$url" | sed 's|https://|http://|g') -- 2.23.0

1 0

[PATCH v2 lkp-tests] lkp-tests/tests: add rpmbuild-pkg to build RPM
by Zhang Yale 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/rpmbuild-pkg | 66 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100755 tests/rpmbuild-pkg diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg new file mode 100755 index 00000000..d70b7c94 --- /dev/null +++ b/tests/rpmbuild-pkg @@ -0,0 +1,66 @@ +#!/bin/bash +# - rpm_repo + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh + +check_vars() +{ + [ -n "$rpm_repo" ] || die "rpm_repo is empty" +} + +benchmark=${rpm_repo##*/} + +RPM_MNT=/initrd/rpmbuild-pkg +mkdir -p "$RPM_MNT" +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} +sync_dest="$RPM_MNT/$pack_to" + +rpmdev-setuptree # generate workspace in $HOME + +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" +} + +request_pkg() +{ + filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + + local dir="SOURCES" + for pkgfile in ${filelist[*]} + do + echo $pkgfile | egrep "\.spec$" && { + dir="SPECS" + } + get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + done +} + +rpmbuild_pkg() +{ + spec_dir=$HOME/rpmbuild/SPECS/$benchmark.spec + #sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $benchmark.spec` + # Install build depends + yum-builddep -y $spec_dir + # Download tar.gz to default path $HOME/rpmbuild/SOURCE + spectool -g -R $spec_dir + # Building rpm or srpm packages + rpmbuild -ba $spec_dir +} + +upload_pkg() +{ + #rpm package will be generated in "$HOME/rpmbuild/SRPMS" and "$HOME/rpmbuild/RPMS" + for file in $(find $HOME/rpmbuild/ -type f -name "*.rpm") + do + upload_one_curl ${file} ${sync_dest} + done +} + +check_vars +request_pkg +rpmbuild_pkg +upload_pkg -- 2.23.0

3 4

[PATCH compass-ci] delimiter: use lkp-tests to upload bisect.log
by Cao Xueliang 09 Dec '20

09 Dec '20

We create a file and put it in "/tmp/lkp/result" dir, the lkp-tests will upload the file when finished the job. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 1 + src/delimiter/constants.rb | 2 ++ src/delimiter/utils.rb | 10 ++-------- 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 7543646..b54be4d 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -81,6 +81,7 @@ class GitBisect if temp[0].include? 'Bisecting' result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} "#{@error_id}" #{@work_dir}` end + Utils.create_bisect_log(@work_dir) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) puts "\nbisect result: #{result}" analyse_result(result) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 5e5406f..7097c6f 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -11,3 +11,5 @@ TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" RESULT_WEBDAV_URL = ENV['RESULT_WEBDAV_URL'] || "http://172.17.0.1:3080" +# The files which are in this dir can be uploaded by lkp-tests +TMP_RESULT_ROOT = '/tmp/lkp/result' diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 60b64a1..b60ada0 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -122,18 +122,12 @@ module Utils return bisect_log_arr end - def create_bisect_log(job_id, git_dir) - log_file = File.join(git_dir, "#{job_id}_bisect.log") + def create_bisect_log(git_dir) + log_file = File.join(TMP_RESULT_ROOT, "bisect.log") log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| log_content.each { |line| f.puts(line) } end - - return log_file - end - - def upload_bisect_log(log_file, dest_dir, access_key) - %x(curl -sSf -T "#{log_file}" #{RESULT_WEBDAV_URL}/#{dest_dir} --cookie "ACCESSKEY=#{access_key}") end end end -- 2.23.0

1 1

[PATCH v1 compass-ci] delimiter: use lkp-tests to upload bisect.log
by Cao Xueliang 09 Dec '20

09 Dec '20

We create a file and put it in "/tmp/lkp/result" dir, the lkp-tests will upload the file when finished the job. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 1 + src/delimiter/constants.rb | 3 ++- src/delimiter/utils.rb | 10 ++-------- 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 7543646..b54be4d 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -81,6 +81,7 @@ class GitBisect if temp[0].include? 'Bisecting' result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} "#{@error_id}" #{@work_dir}` end + Utils.create_bisect_log(@work_dir) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) puts "\nbisect result: #{result}" analyse_result(result) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 5e5406f..c243df5 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -10,4 +10,5 @@ DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" -RESULT_WEBDAV_URL = ENV['RESULT_WEBDAV_URL'] || "http://172.17.0.1:3080" +# The files which are in this dir can be uploaded by lkp-tests +TMP_RESULT_ROOT = ENV['TMP_RESULT_ROOT'] || '/tmp/lkp/result' diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 60b64a1..b60ada0 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -122,18 +122,12 @@ module Utils return bisect_log_arr end - def create_bisect_log(job_id, git_dir) - log_file = File.join(git_dir, "#{job_id}_bisect.log") + def create_bisect_log(git_dir) + log_file = File.join(TMP_RESULT_ROOT, "bisect.log") log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| log_content.each { |line| f.puts(line) } end - - return log_file - end - - def upload_bisect_log(log_file, dest_dir, access_key) - %x(curl -sSf -T "#{log_file}" #{RESULT_WEBDAV_URL}/#{dest_dir} --cookie "ACCESSKEY=#{access_key}") end end end -- 2.23.0

1 0