- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/rpmbuild-pkg: change the way obtaining spec file
by Li Ping 28 Apr '21

28 Apr '21

[why] tests/rpmbuild-pkg: package_name=${upstream_repo##*/} when the package name is inconsistent with the repo name, it will cause the error like: + grep http -rl /root/rpmbuild/SPECS/zstd-test.spec grep: /root/rpmbuild/SPECS/zstd-test.spec: No such file or directory + spectool -g -R /root/rpmbuild/SPECS/zstd-test.spec Can't open '/root/rpmbuild/SPECS/zstd-test.spec': No such file or directory at /usr/bin/spectool line 314. Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 5c694b9d4..3dd75b896 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -49,7 +49,7 @@ init_workspace() download_upstream_repo() { - local url="git://${GIT_SERVER}/openeuler/${upstream_repo}" + local url="git://${GIT_SERVER}/${upstream_dir}/${upstream_repo}" git clone "$url" || die "clone git repo ${package_name} failed: $url" cd "$package_name" || exit @@ -75,14 +75,14 @@ build_rpm() [ -n "$package_name" ] && { # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/"*.spec)" } # Install build depends yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE [ -n "$package_name" ] && { - spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + spectool -g -R "$spec_dir/"*.spec || die "failed to download source file" } # Building rpm or srpm packages rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" -- 2.23.0

1 0

[PATCH v6 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 28 Apr '21

28 Apr '21

To make sure git_repos displayed on the web are right. examples of git_repo: a/AvxToNeon/AvxToNeon 0/caterqiu/001 aur-a/a2png/a2png m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 9e899d9..e305ac6 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -400,6 +400,14 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + + # What does this regular expression want: + # 1. a three-segment structure "xxx/xxx/xxx" + # 2. there can be '-' or '_' in every segment, but can't be at first or last of the segment. + # 3. the first segment can have lowercase of letters or numbers in it. + # 4. the other two segments can have letters(lowercase or uppercase) or numbers in it. + next unless r['git_repo'] =~ %r{^([a-z0-9]([a-z0-9\-_]*[a-z0-9])*(/[a-zA-Z0-9][\w\-]*[a-zA-Z0-9]){2})$} + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH lkp-tests] jobs/send-email.yaml: change the default value.
by Li Ping 28 Apr '21

28 Apr '21

the default value of email subject should not be job_failed, and there is already a send-email-on-fail.yaml. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/send-email.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/send-email.yaml b/jobs/send-email.yaml index 8f8b48b96..2689634c2 100644 --- a/jobs/send-email.yaml +++ b/jobs/send-email.yaml @@ -3,4 +3,4 @@ # values of $subject are limited to job_failed/job_debug/borrow_success/borrow_failed. send_email: - subject: job_failed + subject: -- 2.23.0

1 0

[PATCH compass-ci 2/2] kibana/dashboard_import: import kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can import kibana dashboards by json files. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_import | 45 +++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/kibana/dashboard_import diff --git a/container/kibana/dashboard_import b/container/kibana/dashboard_import new file mode 100755 index 0000000..670fa3f --- /dev/null +++ b/container/kibana/dashboard_import @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'pathname' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +files = [] +ARGV.each do |f| + files >> f +end + +if files.empty? + profile = Pathname.new(File.dirname(__FILE__)).realpath + files << profile + 'data.json' + files << profile + 'resource.json' + files << profile + 'all.json' +end +puts files + +cmd = "curl -XPOST -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/import" +cmd += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + +files.each do |f| + new_cmd = cmd + new_cmd += " -d @#{f}" + puts new_cmd + system new_cmd +end -- 2.23.0

2 1

[PATCH compass-ci] src/lib/web_backend.rb: modify parameter list for web compare
by Li Yuanchao 28 Apr '21

28 Apr '21

The parameter list for web compare need to change, as tbox_groups have $user at the back. Suite and tbox_group whose count < 2 need not to show, cause have no one to compare. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index da3c3c5..a9ab091 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -45,16 +45,11 @@ def es_count(query) ES_CLIENT.count(index: 'jobs*', body: query)['count'] end -# "vm-2p8g--212" remove "--212" -# "vm-2p8g--zzz" remove "--zzz" -# "vm-git-bisect" don't remove "-bisect" +# delete $user after '--' or '.' or '~' def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ - index = r.index('--') || r.rindex('-') - r = r[0, index] - end + r = r.gsub(/(--|\.|~).*$/, '') result.add r end result.to_a @@ -70,10 +65,10 @@ def all_suite size: 0 } es_result = es_query(query)['aggregations']['all_suite']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - - es_result + result_second = [] + es_result.each { |e| result_second << e if e['doc_count'] > 1 } + result_second.sort_by! { |h| h['doc_count'] } + result_second.reverse!.map! { |x| x['key'] } end def all_tbox_group @@ -86,10 +81,12 @@ def all_tbox_group size: 0 } es_result = es_query(query)['aggregations']['all_tbox_group']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } + result_second = [] + es_result.each { |e| result_second << e if e['doc_count'] > 1 } + result_second.sort_by! { |h| h['doc_count'] } + result_second.reverse!.map! { |x| x['key'] } - filter_tbox_group(es_result) + filter_tbox_group(result_second) end def compare_candidates_body -- 2.23.0

2 1

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: add field
by Lu Weitao 28 Apr '21

28 Apr '21

add nr_run(integer), author_email(keyword) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 67b1ed14..0c2fc62d 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -148,6 +148,12 @@ else "upstream_commit": { "type": "keyword" }, + "base_commit": { + "type": "keyword" + }, + "nr_run": { + "type": "integer" + }, "my_email": { "type": "keyword" }, -- 2.23.0

2 2

[PATCH compass-ci 1/2] kibana/dashboard_export: exporting kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can export kibana dashboards by ids. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_export | 37 +++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/kibana/dashboard_export diff --git a/container/kibana/dashboard_export b/container/kibana/dashboard_export new file mode 100755 index 0000000..d6e0054 --- /dev/null +++ b/container/kibana/dashboard_export @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +dashboard_ids = [] +ARGV.each do |i| + dashboard_ids << i +end + +dashboard_ids = %w[data resource all] if dashboard_ids.empty? + +cmd = "curl -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/export?dashboard=" + +dashboard_ids.each do |id| + new_cmd = cmd + new_cmd += "#{id} >> #{id}.json" + puts new_cmd + system new_cmd +end -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] iso2rootfs: some optimize
by Yu Chuan 27 Apr '21

27 Apr '21

1. change dailybuild result rootfs dir name why: - make the dailybuild rootfs directory more meaningful. how: - before: /srv/os/${iso_os}/${iso_arch}/${iso_version}-${TIMESTAMP} - after : /srv/os/${iso_os}/${iso_arch}/${iso_version}-dailybuild-${TIMESTAMP} 2. change sha256sum file why: - unify the file name, then easier to statistics. how: - file name: - before: ${ROOTFS_DES_DIR}/${iso_version}-latest.sha256sum - after : ${ROOTFS_DES_DIR}/SHA256SUM Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 06c8565110ab..ca972b2399f6 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -318,6 +318,8 @@ download_compass_ci() config_rootfs_dir() { ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") + [ -n "${dailybuild_iso_url_file}" ] && ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-dailybuild-$(date "+%Y-%m-%d-%H-%M-%S") + [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -592,7 +594,7 @@ update_dailybuild_soft_link() [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" - cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}" + cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}/SHA256SUM" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] install-iso: fix some bugs
by Yu Chuan 27 Apr '21

27 Apr '21

[What Bugs] 1. optimize name: reboot.cgz => reboot-after-systemd-logind.cgz 2. nfs mount result dir: - before: ${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} - after : ${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} 3. exit directory before umount Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 6 +++--- tests/install-iso-sync-rootfs | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c51b3784b5aa..481b99e35df4 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -196,8 +196,8 @@ submit_install_iso_firstboot_yaml() custom_ipxe="#!ipxe\\\\n" custom_ipxe="${custom_ipxe}\\\\ndhcp" custom_ipxe="${custom_ipxe}\\\\ninitrd http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/initrd.lkp" - custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot.cgz" - custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot.cgz local rw" + custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot-after-systemd-logind.cgz" + custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot-after-systemd-logind.cgz local rw" custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" $firstboot_yaml @@ -225,7 +225,7 @@ submit_install_iso_sync_rootfs_yaml() install-iso-sync-rootfs.NFS_ROOT_DIR=${NFS_ROOT_DIR} \ install-iso-sync-rootfs.ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ install-iso-sync-rootfs.VG_NAME=${VG_NAME} \ - testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" + testbox=$testbox || die "submit job failed: ${sync_rootfs_yaml}" } main() diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index 34096229ccfb..b74c7bed2044 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -59,7 +59,7 @@ sync_rootfs_to_nfs_server() local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} mkdir -p ${nfs_rootfs_mount_point} - mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" # activate the lvm vgchange -ay @@ -78,7 +78,7 @@ sync_rootfs_to_nfs_server() cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_SYNC_DIRS[@]}" # umount nfs rootfs - umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + cd / && umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" # umount local logical volume umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] install-iso-pre: change the logic of ks generate and upload
by Yu Chuan 27 Apr '21

27 Apr '21

[Why] 1. 'key difference' of ks for different physical machine: - Choice Install Source - Choice Install Partition - Choice working Nic 2. 'other ks content' for different os is different. so the change is as following: before: 'other ks content' is all same. after : add a httpd dir, which stores the 'other ks content' for different os, then they only need generate the 'key difference' related to current physical machine, 'other ks content' can download from httpd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 36 ++---------------------------------- 1 file changed, 2 insertions(+), 34 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c89895a994cc..c51b3784b5aa 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -112,41 +112,9 @@ generate_ks() bootloader --location=none logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting - # Root password - rootpw --iscrypted ${secrets_ROOTFS_ROOT_PASSWD} - - # Reboot after installation - reboot --eject - - # Use text mode install - text - - # Keyboard layouts - keyboard --vckeymap=us --xlayouts='us' - - # System language - lang en_US.UTF-8 - - # Do not configure the X Window System - skipx - - # System services - services --enabled="chronyd" - - # System timezone - timezone Asia/Shanghai --isUtc - - %packages - @^minimal-environment - - %end - - %anaconda - pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty - pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok - pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty - %end EOF + + curl -s http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/install/ks/template/${iso_os}-${iso_arch}-${iso_version}.ks >> $ks_file } upload_ks() -- 2.23.0

1 0