- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci] container/result-webdav: adjust the value of second_dir
by Cui Yili 02 Dec '20

02 Dec '20

[why] 1. when the uri starts with "//", etc., the previous regular expression did not get the second level directory correctly. 2. when the uploaded data package does not meet the format of "xxxx_yyyymmdd.cgz", uploading is also allowed, but no symbolic link. [how] 1. modify the variable second_dir regular expression. 2. make conditional judgments on link_name. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index c880522..98382bd 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,17 +54,21 @@ http { path = string.match(uri,"%g*/") upload_file = string.match(uri, ".+/([^/]*%.%w+)$") - second_dir = string.match(uri, "/.+/(.*)/.+") + second_dir = string.match(uri, "/*%w+/(.-)/.+") - if (second_dir == "pkg") + if (second_dir == "pkg" or second_dir == "build-pkg") then - link_name = "latest.cgz" + link_name = "latest" else - link_name = string.match(upload_file, "(.*)_%d+%.cgz")..".cgz" + link_name = string.match(upload_file, "(.*)_%d+%.cgz") + end + + if (link_name ~= '') + then + link_name = link_name..".cgz" + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) end - - root_patch = "/srv/"..path - io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) } } -- 2.23.0

1 0

[PATCH v4 compass-ci] scheduler: fix the paths to find cluster config file
by Ren Wen 02 Dec '20

02 Dec '20

Find cluster config file at *lab-z9/cluster/* first, *lkp-tests/cluster/* otherwise. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 3 +++ src/scheduler/submit_job.cr | 14 +++++++++----- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 418e7ff..b2ea831 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -21,6 +21,9 @@ INITRD_HTTP_PORT = (ENV.has_key?("INITRD_HTTP_PORT") ? ENV["INITRD_HTTP_PORT"] : OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1") OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 +CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") +LAB_REPO = "lab-z9" + SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 74ae4b2..a9eb3a7 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -15,7 +15,7 @@ class Sched else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, - job.os_arch) + job.os_arch).not_nil! return submit_cluster_job(job, cluster_config) end rescue ex @@ -147,12 +147,16 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) + + cluster_file_paths = [ + Path.new(CCI_REPOS, LAB_REPO, "cluster", cluster_file), + Path.new(lkp_src, "cluster", cluster_file) + ] + cluster_file_paths.each do |f| + return YAML.parse(File.read(f)) if File.file?(f) + end end def get_commit_date(job) -- 2.23.0

1 0

[PATCH lkp-tests] scheduler: fix: $server not $direct_server_ips
by Zhang Yu 02 Dec '20

02 Dec '20

[why] Now the server ip is defined by scheduler is $server not $direct_server_ips [how] Change the $direct_server_ips to $server Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 106ee0b9..1d424aaa 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -19,7 +19,7 @@ # db_driver : "${mysql_user:=root}" -: "${mysql_host:=$direct_server_ips}" +: "${mysql_host:=$server}" : "${mysql_port:=$mysql_port}" : "${mysql_db:=sysbench_1}" : "${mysql_password:=$mysql_password}" -- 2.23.0

1 0

[PATCH lab-z9] hosts: add taishan200-2280-2s48p-256g--a12 host-info
by Xiao Shenwei 02 Dec '20

02 Dec '20

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a12 | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a12 diff --git a/hosts/taishan200-2280-2s48p-256g--a12 b/hosts/taishan200-2280-2s48p-256g--a12 new file mode 100644 index 0000000..ce014c1 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a12 @@ -0,0 +1,22 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-35000c500c276c347 + - /dev/disk/by-id/scsi-35000c500c276eb87 + - /dev/disk/by-id/scsi-35000c500c276b6ff +rootfs_partition: + - /dev/disk/by-id/scsi-35000c500c276f47b +mac_addr: + - 44:67:47:97:22:37 + - 44:67:47:97:22:38 + - 44:67:47:97:22:39 + - 44:67:47:97:22:3a + - 44:67:47:97:22:3b + - 44:67:47:97:22:3c + - 44:67:47:97:22:3d + - 44:67:47:97:22:3e +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.2 -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] sbin: load PKGBUILD.src content
by Liu Shaofei 02 Dec '20

02 Dec '20

The contents of PKGBUILD.src is the following example: source=( "http://172.168.131.113:8800/initrd/build-tar/ansible/ansible-2.9.4.tar.gz" ) Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 6f8b95af..498b4872 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -895,6 +895,7 @@ update_pkgver() { sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_safe "PKGBUILD.src" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3699,6 +3700,7 @@ else fi source_buildfile "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_buildfile PKGBUILD.src fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

[PATCH v2 lkp-tests] spec/submit: add job yaml for changing multi-args style
by Bai Jing 02 Dec '20

02 Dec '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 6 ++--- .../unixbench-spec-1-300-pipe.yaml | 25 +++++++++++++++++++ ...unixbench-spec-1-300-whetstone-double.yaml | 8 +++--- .../unixbench-spec-3-300-shell1.yaml | 8 +++--- .../unixbench-spec-3-300-shell8.yaml | 25 +++++++++++++++++++ 5 files changed, 61 insertions(+), 11 deletions(-) create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml index e17322b4..34fb4996 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -2,8 +2,8 @@ suite: unixbench runtime: 300s -nr_task test: - - 1 whetstone-double - - 3 shell1 +nr_task|test: + - 1|pipe + - 3|shell8 unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml new file mode 100644 index 00000000..44be6b10 --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: pipe +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: pipe + nr_task: '1' + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml index 9896bcef..78f79fa7 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '1' @@ -13,7 +13,7 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +21,5 @@ pp: runtime: 300 test: whetstone-double nr_task: '1' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml index 912d10ee..afe5a953 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '3' @@ -13,7 +13,7 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +21,5 @@ pp: runtime: 300 test: shell1 nr_task: '3' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml new file mode 100644 index 00000000..fa58377c --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell8 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell8 + nr_task: '3' + cpufreq_governor: +monitors: -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: 修改名字为如何登录测试机调测任务.md并增加和修改内容
by Zhang Yale 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...03\346\265\213\344\273\273\345\212\241.md" | 69 ++++++++++++++----- 1 file changed, 53 insertions(+), 16 deletions(-) rename doc/manual/how-to-log-in-the-machine.md => "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" (55%) diff --git a/doc/manual/how-to-log-in-the-machine.md "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" similarity index 55% rename from doc/manual/how-to-log-in-the-machine.md rename to "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index fefd718..f4f33a3 100644 --- a/doc/manual/how-to-log-in-the-machine.md +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -1,17 +1,20 @@ -这篇文档将告诉你如何登陆测试环境 +这篇文档将告诉你如何登陆测试环境去调测任务 -# 1. 前提条件 -请先学习: +# 1. 前提条件： +## 请先学习: * [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 * [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 * [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 # 2. 操作方法 -## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： -```shell +## 2.1 如何修改yaml? +### 在测试机运行script之前进入测试机调测任务，可以在job.yaml里加上sshd和sleep字段，测试机在运行脚本之前就sleep, 并免密登录进去，可以手动输入命令或脚本进行调试，以host-inoyaml为例： + +```yaml suite: host-info category: functional + pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -19,6 +22,7 @@ nil end %> + sshd: # sleep at the bottom sleep: 1h @@ -26,19 +30,51 @@ host-info: ``` -## 2.2 以下有2种方式可以登录到测试机： -### 第一种：使用submit -m -c的方式： - 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 - 命令：submit -m -c host-info.yaml - 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： +### 在测试任务运行失败时进入测试机调测任务，可以在job.yaml里加上pub_key和on_fail字段，并在on_fail下加上sshd和sleep字段, 测试任务失败后，测试机保留并免密登录进去，以方便手动调试，以iperf.yaml为例： + +```yaml + suite: iperf + testcase: iperf + category: benchmark + + runtime: 300s + + cluster: cs-localhost + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + if role server: + iperf-server: + + if role client: + iperf: + protocol: + - tcp + - udp + + on_fail: + sshd: + sleep: 1000s +``` +## 如何免密登录测试机? +### submit时加上-m -c，该方法不用查看邮件登录，自动免密登录到测试机： + + 命令：submit -m -c iperf.yaml + 效果如下： ```shell - hi6325@account-vm ~% submit -m -c atomic.yaml - submit atomic.yaml, got job_id=crystal.146528 + hi6325@account-vm ~% submit -m -c iperf.yaml + submit iperf.yaml, got job_id=crystal.146528 query=>{"job_id":["crystal.146528"]} connect to ws://localhost:11310/filter - {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "result_root": "/srv/result/iperf/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" {"job_id": "crystal.146528", "job_state": "boot"} {"job_id": "crystal.146528", "job_state": "download"} "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} @@ -48,9 +84,10 @@ root@vm-2p8g ~# ``` -### 第二种：根据邮件信息使用ssh方式登录测试机： - 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 - 命令： submit host-info.yaml +### 直接submit或者submit只加-m参数，可在收到的邮件里，按照邮件提示手动免密登录到测试机调测： + + 命令：submit iperf.yaml + 任务执行完成后，系统发送邮件内容如下： ```shell -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] send-internet-mail: update start
by Luan Shengde 02 Dec '20

02 Dec '20

attach MAILDIR to the container store mail files that have been sent - MAILDIR/sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 483f01a..d6e4e25 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -12,25 +12,28 @@ names = Set.new %w[ ROBOT_EMAIL_ADDRESS ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT + MAILDIR ] defaults = relevant_defaults(names) defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] ||= '/srv/cci/Maildir/.compass-ci' +defaults['HOST_SERVER'] = ENV['HOSTNAME'] env = docker_env(defaults) -cmd = %W[ +cmd = %w[ docker run --restart=always --name=send-internet-mail -d ] + env + %W[ -p #{defaults['SEND_MAIL_PORT']}:#{defaults['SEND_MAIL_PORT']} + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']} -v /etc/localtime:/etc/localtime:ro -w #{defaults['CCI_SRC']}/container/send-internet-mail - --log-driver json-file send-internet-mail ] -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] send-internet-mail/mail-post: call to store email
by Luan Shengde 02 Dec '20

02 Dec '20

call function store_email to store email to directory sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 0b1f94a..df4d255 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -51,5 +51,11 @@ def send_mail(mail_info) to mail_info['to'] body mail_info['body'] end + mail.deliver! + + return if ENV['SEND_MAIL_PORT'].to_s != '49000' + return if ENV['HOST_SERVER'] != 'z9' + + store_email(mail) end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix upload in container
by Wang Yong 02 Dec '20

02 Dec '20

docker can not use mount operation, so now use curl to upload build-pkg cgz file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef0355..6e435a6ad 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -18,6 +18,8 @@ ## See: https://wiki.archlinux.org/index.php/Makepkg . $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/env.sh +. $LKP_SRC/distro/common check_vars() { @@ -29,6 +31,13 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +mount_pkg_mnt() +{ + [ -n "$LKP_SERVER" ] && { + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + } +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -39,9 +48,7 @@ mount_dest() PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" - [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" - } + is_docker || mount_pkg_mnt } get_pkgfile() @@ -117,10 +124,15 @@ build_source_pkg() } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 - [ "$?" == 0 ] && create_softlink + [ "$?" == 0 ] || exit 1 + create_softlink + + is_docker || exit 0 + upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } check_vars -- 2.23.0

1 0