- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] container/web-backend: add api for group jobs stats count
by Li Yuanchao 17 Mar '21

17 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..131cff0 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_group_jobs_table?suite=iperf&dimension=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - dimension +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming" : { +# "all" : $all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fali +# }, +# "chenqun" : { +# "all" : $all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fali +# }, +# ... +# } +get '/get_group_jobs_table' do + group_jobs_table(params) +end -- 2.23.0

2 2

[PATCH compass-ci] sparrow/3-code/dev-env: set $lab default value
by Liu Yinsi 17 Mar '21

17 Mar '21

before: $lab always is nolab after: if env $lab already exists, use env value. it can make code more flexible. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 6072ce8..e182b65 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -4,6 +4,7 @@ [ -z $server_ip ] && server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') server_name=$(hostname | cut -f1 -d.) +: ${lab:=nolab} : ${sched_host:=$server_ip} : ${sched_port:=3000} : ${ES_HOST:=$server_ip} @@ -38,7 +39,7 @@ mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port -lab: nolab +lab: $lab EOF mkdir -p /etc/compass-ci/service -- 2.23.0

1 0

[PATCH lkp-tests] tests/deploy-ci: delete env GITCACHE_HOST and GITCACHE_PORT
by Liu Yinsi 17 Mar '21

17 Mar '21

[why] gitcache service cannot be used to maintain branches when build compass-ci/container/crystal-shards images, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/deploy-cci | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 8e6acbac4..6f97d7151 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -6,8 +6,6 @@ export_env() { export initrd_http_host=$INITRD_HTTP_HOST export initrd_http_port=$INITRD_HTTP_PORT - export gitcache_host=$GITCACHE_HOST - export gitcache_port=$GITCACHE_PORT } deploy() -- 2.23.0

1 0

[PATCH v2 compass-ci] result-webdav-webdav/nginx.conf: do not overwrite the uploaded config with the same name
by Lin Jiaxin 17 Mar '21

17 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/result-webdav/nginx.conf | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index cc21926..e0e0ee2 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -27,6 +27,22 @@ http { dav_methods PUT MKCOL; dav_access user:rw group:rw all:rw; + set $config_upload_condition "false"; + if ( $uri ~ "^/cci/build-config.*" ) + { + set $config_upload_condition "true"; + } + + if ( -e $request_filename ) + { + set $config_upload_condition "${config_upload_condition}+true"; + } + + if ( $config_upload_condition = "true+true" ) + { + return 200; + } + access_by_lua_block { function split(str, reps) local resultstrlist = {} -- 2.23.0

1 0

[PATCH v2 lkp-tests] rpmbuild for openeuler
by Li Ping 17 Mar '21

17 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 2 +- tests/rpmbuild-pkg | 47 +++++++++++++++++++++++++----------------- 2 files changed, 29 insertions(+), 20 deletions(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 53e145b3b..a70590afe 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,5 @@ suite: rpmbuild-pkg category: functional -rpm_repo: +upstream_repo: rpmbuild-pkg: diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..86b775334 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,12 +1,16 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo +# - compat_os . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +: "${compat_os:=bud-openeuler}" + +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} +rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/${os_arch}/Packages" +src_rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/source/Packages" init_workspace() { @@ -14,21 +18,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } @@ -50,19 +49,29 @@ build_rpm() show_rpm_files() { find ${HOME}/rpmbuild/RPMS -type f -name "*.rpm" +} + +show_src_rpm_files() +{ find ${HOME}/rpmbuild/SRPMS -type f -name "*.rpm" } upload_rpm_pkg() { - local file - for file in $(show_rpm_files) + local rpm_file + for rpm_file in $(show_rpm_files) + do + upload_one_curl ${rpm_file} ${rpm_dest} + done + + local src_rpm_file + for src_rpm_file in $(show_src_rpm_files) do - upload_one_curl ${file} ${sync_dest} + upload_one_curl ${src_rpm_file} ${src_rpm_dest} done } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

1 1

[PATCH v3 lkp-tests] rpmbuild for openeuler
by Li Ping 17 Mar '21

17 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 2 +- tests/rpmbuild-pkg | 47 +++++++++++++++++++++++++----------------- 2 files changed, 29 insertions(+), 20 deletions(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 53e145b3b..a70590afe 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,5 @@ suite: rpmbuild-pkg category: functional -rpm_repo: +upstream_repo: rpmbuild-pkg: diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..256e836bf 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,12 +1,16 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo +# - compat_os . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +: "${compat_os:=budding-openeuler}" + +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} +rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/${os_arch}/Packages" +src_rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/source/Packages" init_workspace() { @@ -14,21 +18,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } @@ -50,19 +49,29 @@ build_rpm() show_rpm_files() { find ${HOME}/rpmbuild/RPMS -type f -name "*.rpm" +} + +show_src_rpm_files() +{ find ${HOME}/rpmbuild/SRPMS -type f -name "*.rpm" } upload_rpm_pkg() { - local file - for file in $(show_rpm_files) + local rpm_file + for rpm_file in $(show_rpm_files) + do + upload_one_curl ${rpm_file} ${rpm_dest} + done + + local src_rpm_file + for src_rpm_file in $(show_src_rpm_files) do - upload_one_curl ${file} ${sync_dest} + upload_one_curl ${src_rpm_file} ${src_rpm_dest} done } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

1 0

[PATCH lkp-tests] add new error type
by Bai Jing 17 Mar '21

17 Mar '21

[input] Error: Module or Group 'convert' does not exist [output] error.nor-exit-module-or-group.convert: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 4c65c6bb3..97cf24d7f 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -18,6 +18,11 @@ while (line = STDIN.gets) puts "unknown-repo.#{repo}: 1" puts "unknown-repo.#{repo}.message: #{line}" + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + puts "error.not-exist-module-or-group.#{repo}: 1" + puts "error.not-exist-module-or-group.#{repo}.message: #{line}" # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" -- 2.23.0

1 0

[PATCH compass-ci] container/logging-es: remove redundant code
by Liu Yinsi 17 Mar '21

17 Mar '21

[why] these deleted code optimize the bulk write performance, but affect the real-time performance of data. At now, our fluentd architecture already been optimized, the bulk write is no longer a bottleneck, so delete codes. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/start | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/container/logging-es/start b/container/logging-es/start index efd89de..9be7a5b 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -19,14 +19,3 @@ cmd=( ) "${cmd[@]}" - -# set index -check_service_ready 9202 -curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' -{ - "index": { - "refresh_interval": "120s", - "translog.durability": "async", - "translog.sync_interval": "120s", - "translog.flush_threshold_size": "1024mb"} -}' -- 2.23.0

1 0

[PATCH compass-ci] dracut-initrd: add xz-utils
by Wang Chenglong 17 Mar '21

17 Mar '21

[Why] wcl@crystal ~/compass-ci/container/dracut-initrd% ./run /srv/os/centos/x86_64/6.10/lib/modules/3.10.0-957.1.3.el7.x86_64 dracut: Executing: /usr/bin/dracut -o iscsi --omit-drivers iscsi --force --kver 3.10.0-957.1.3.el7.x86_64 -k /lib/modules/3.10.0-957.1.3.el7.x86_64 /boot/initramfs.lkp-3.10.0-957.1.3.el7.x86_64.img dracut: Kernel modules are compressed with xz, but xz is not available. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 616183f..b8eb6f5 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -6,7 +6,7 @@ export DEBIAN_FRONTEND=noninteractive apt-get update apt-get install -y --fix-missing --no-install-recommends -qq apt-utils \ -nfs-common netbase cifs-utils kmod rsync dracut dracut-network \ +nfs-common netbase cifs-utils kmod rsync dracut dracut-network xz-utils \ dracut-config-generic apt-get clean -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: add secrets_info.yaml
by Wei Jihui 17 Mar '21

17 Mar '21

secrets_info.yaml is for user use secret field Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/secrets_info.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/secrets_info.yaml diff --git a/jobs/secrets_info.yaml b/jobs/secrets_info.yaml new file mode 100644 index 00000000..4026b3b7 --- /dev/null +++ b/jobs/secrets_info.yaml @@ -0,0 +1,6 @@ +# This file is for user add secret field, so that it's not visible from outside(job.yaml), but can be used at testbox. +# Usage: submit job.yaml -i secrets_info.yaml +# Use at testbox: testbox has env with: "secrets_" + secret field, such as $secrets_my_token + +secrets: + my_token: {{my_token}} -- 2.23.0

1 0