- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] params_group.rb: delete $user if tbox_group has
by Li Yuanchao 02 Dec '20

02 Dec '20

Some tbox_groups have $user in es. Such as vm-2p8g--$user or vm-2p8g.$user. But their config is the same, they should be compared together. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/params_group.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index cde40af..57cd5c1 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -67,10 +67,15 @@ def filter_groups(groups) end end +def get_tbox_group(tbox_group) + return tbox_group.gsub(/(--|\.).*$/, '') +end + def get_all_params(job) all_params = {} job.each_key do |param| all_params[param] = job[param] if COMMON_PARAMS.include?(param) + all_params[param] = get_tbox_group(job[param]) if param == 'tbox_group' next unless param == 'pp' pp_params = get_pp_params(job[param]) -- 2.23.0

2 3

[PATCH compass-ci 2/2] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..a897b57 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,31 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each_index do |index| + testboxes[index] = buckets[index]['key'] + end + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/web_backend: add a interface /get_testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

It would return a testbox list to web request Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index d68e9f2..11ba604 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -103,3 +103,10 @@ post '/compare_template' do request.body.rewind # in case someone already read it compare_template(request.body.read) end + +# GET /get_testboxes +# Response: +# - { total: total, testboxes[ 'testbox1', 'testbox2', ... ] } +get '/get_testboxes' do + query_testboxes +end -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 02 Dec '20

02 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..c788607 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,16 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic configurations, such as arch, nr_node, nr_cpu, memory + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: memory size of the machine. + +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query.rb: add traverse es data of function
by Cui Yili 02 Dec '20

02 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index c027aa9..4c99023 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -11,6 +11,7 @@ class ESQuery PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i def initialize(host = HOST, port = PORT, index: 'jobs') @index = index + @scroll_id = "" @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end @@ -33,6 +34,23 @@ class ESQuery @client.search index: 'jobs*', body: query end + def traverse_field(size) + if @scroll_id.empty? + query = { + query: { + bool: { + must: {match_all: {} } + } + }, size: size + } + result = @client.search index: @index, scroll: '10m', body: query + @scroll_id = result['_scroll_id'] + return result + else + @client.scroll scroll: '10m', scroll_id: @scroll_id + end + end + def query_by_id(id) @client.get_source({ index: @index, type: '_doc', id: id }) rescue Elasticsearch::Transport::Transport::Errors::NotFound -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] tests: replace *.tar.* address with local address
by Liu Shaofei 02 Dec '20

02 Dec '20

When the address of source field is the format of "*.tar.*", we need to replace it with local address. The following example: source=("https://github.com/azukiapp/${pkgname}/archive/v${pkgver}.tar.gz") => source=("http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/v${pkgver}.tar.gz") Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef035..cde2f0ed 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -98,6 +98,26 @@ replace_source() sed -i "s|^source=[^)]*|$1|g" "$2" } +# replace *.tar.* address from source field in PKGBUILD +replace_tar_source() +{ + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar' && { + local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" + + # check whether local server exists *.tar.* file + wget -q --spider "${local_url}" + [ "$?" == 0 ] || continue + + url="\"${local_url}\"" + } + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src +} + build_source_pkg() { local repo_dir="" @@ -116,6 +136,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } + replace_tar_source + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -- 2.23.0

2 2

[PATCH compass-ci] params_group.rb: delete $user if tbox_group has
by Li Yuanchao 02 Dec '20

02 Dec '20

Some tbox_groups have $user in es. Such as vm-2p8g--$user or vm-2p8g.$user. But their config is the same, they should be compared together. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/params_group.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index cde40af..6d5c47b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -67,10 +67,17 @@ def filter_groups(groups) end end +def get_tbox_group(tbox_group) + tbox_group = tbox_group.split('--')[0] if tbox_group.include?('--') + tbox_group = tbox_group.split('.')[0] if tbox_group.include?('.') + return tbox_group +end + def get_all_params(job) all_params = {} job.each_key do |param| all_params[param] = job[param] if COMMON_PARAMS.include?(param) + all_params[param] = get_tbox_group(job[param]) if param == 'tbox_group' next unless param == 'pp' pp_params = get_pp_params(job[param]) -- 2.23.0

3 4

[PATCH compass-ci] sched: no need to check kernel config in initialized? function
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7758e78..2a811cb 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,8 +146,8 @@ class Job set_result_root() set_result_service() set_os_mount() - set_kernel() set_depends_initrd() + set_kernel() set_initrds_uri() set_lkp_server() set_sshr_info() @@ -345,7 +345,11 @@ class Job "SCHED_HOST", "SCHED_PORT"] - initialized_keys -= ["my_uuid"] + initialized_keys -= ["my_uuid", + "kernel_version", + "kernel_uri", + "kernel_params", + "linux_vmlinuz_path"] initialized_keys.each do |key| return false unless @hash.has_key?(key) -- 2.23.0

1 0

[PATCH compass-ci] auto_submit.yaml: change git_repo
by Li Yuanchao 02 Dec '20

02 Dec '20

To keep style the same Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index f741b79..e0f52ba 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,6 +1,6 @@ -AvxToNeon/AvxToNeon: +a/AvxToNeon/AvxToNeon: - testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml -compass-ci/compass-ci: +c/compass-ci/compass-ci: - testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=vm-2p8g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/start: add /etc/localtime
by Cui Yili 02 Dec '20

02 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index 01141c3..d7b1097 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -13,6 +13,7 @@ cmd=( --restart=always -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ result-webdav -- 2.23.0

1 0

[PATCH compass-ci] answerback-email: add execute tool limit
by Luan Shengde 02 Dec '20

02 Dec '20

limit the scope of executing answerback-email to 'z9' [why] when call es_client to store the account data, the data will be stored in the es installed at the localhost. It will cause validation to fail when submit a job if the data is not at the server 'z9'. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 33 +++++++++++++++++--- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 825d815..77a2c1f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -187,11 +187,14 @@ def apply_account(my_info, conf_info) end def check_my_email(my_info) + return true if my_info['my_email'] + message = "No email address specified\n" message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' + puts message - raise message if my_info['my_email'].nil? + return false end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -219,13 +222,34 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + return true if ENV['HOSTNAME'] == 'crystal.ci' + + message = 'please run the tool on z9 server' + + puts message + return false +end + +def check_my_name_exist(my_info) + return true if my_info['my_name'] + + message = 'No my_name found, please use -n to add one' + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) - check_my_email(my_info) + puts 'check_erver' + return unless check_server + return unless check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) - message = 'No my_name found, please use -n to add one' - raise message if my_info['my_name'].nil? + puts 'check_my_name' + return unless check_my_name_exist(my_info) account_info = apply_account(my_info, conf_info) build_my_info_from_account_info(my_info, account_info, conf_info) @@ -241,6 +265,7 @@ def send_mail(my_info, account_info, conf_info) else build_message(my_info['my_email'], account_info) end + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -- 2.23.0

1 1

[PATCH v4 compass-ci] container/result-webdav: adjust the value of second_dir
by Cui Yili 02 Dec '20

02 Dec '20

[why] 1. when the uri starts with "//", etc., the previous regular expression did not get the second level directory correctly. 2. when the uploaded data package does not meet the format of "xxxx_yyyymmdd.cgz", uploading is also allowed, but no symbolic link. [how] 1. modify the variable second_dir regular expression. 2. make conditional judgments on link_name. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index c880522..98382bd 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,17 +54,21 @@ http { path = string.match(uri,"%g*/") upload_file = string.match(uri, ".+/([^/]*%.%w+)$") - second_dir = string.match(uri, "/.+/(.*)/.+") + second_dir = string.match(uri, "/*%w+/(.-)/.+") - if (second_dir == "pkg") + if (second_dir == "pkg" or second_dir == "build-pkg") then - link_name = "latest.cgz" + link_name = "latest" else - link_name = string.match(upload_file, "(.*)_%d+%.cgz")..".cgz" + link_name = string.match(upload_file, "(.*)_%d+%.cgz") + end + + if (link_name ~= '') + then + link_name = link_name..".cgz" + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) end - - root_patch = "/srv/"..path - io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) } } -- 2.23.0

1 0

[PATCH v4 compass-ci] scheduler: fix the paths to find cluster config file
by Ren Wen 02 Dec '20

02 Dec '20

Find cluster config file at *lab-z9/cluster/* first, *lkp-tests/cluster/* otherwise. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 3 +++ src/scheduler/submit_job.cr | 14 +++++++++----- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 418e7ff..b2ea831 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -21,6 +21,9 @@ INITRD_HTTP_PORT = (ENV.has_key?("INITRD_HTTP_PORT") ? ENV["INITRD_HTTP_PORT"] : OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1") OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 +CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") +LAB_REPO = "lab-z9" + SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 74ae4b2..a9eb3a7 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -15,7 +15,7 @@ class Sched else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, - job.os_arch) + job.os_arch).not_nil! return submit_cluster_job(job, cluster_config) end rescue ex @@ -147,12 +147,16 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) + + cluster_file_paths = [ + Path.new(CCI_REPOS, LAB_REPO, "cluster", cluster_file), + Path.new(lkp_src, "cluster", cluster_file) + ] + cluster_file_paths.each do |f| + return YAML.parse(File.read(f)) if File.file?(f) + end end def get_commit_date(job) -- 2.23.0

1 0

[PATCH lkp-tests] scheduler: fix: $server not $direct_server_ips
by Zhang Yu 02 Dec '20

02 Dec '20

[why] Now the server ip is defined by scheduler is $server not $direct_server_ips [how] Change the $direct_server_ips to $server Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 106ee0b9..1d424aaa 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -19,7 +19,7 @@ # db_driver : "${mysql_user:=root}" -: "${mysql_host:=$direct_server_ips}" +: "${mysql_host:=$server}" : "${mysql_port:=$mysql_port}" : "${mysql_db:=sysbench_1}" : "${mysql_password:=$mysql_password}" -- 2.23.0

1 0

[PATCH lab-z9] hosts: add taishan200-2280-2s48p-256g--a12 host-info
by Xiao Shenwei 02 Dec '20

02 Dec '20

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a12 | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a12 diff --git a/hosts/taishan200-2280-2s48p-256g--a12 b/hosts/taishan200-2280-2s48p-256g--a12 new file mode 100644 index 0000000..ce014c1 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a12 @@ -0,0 +1,22 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-35000c500c276c347 + - /dev/disk/by-id/scsi-35000c500c276eb87 + - /dev/disk/by-id/scsi-35000c500c276b6ff +rootfs_partition: + - /dev/disk/by-id/scsi-35000c500c276f47b +mac_addr: + - 44:67:47:97:22:37 + - 44:67:47:97:22:38 + - 44:67:47:97:22:39 + - 44:67:47:97:22:3a + - 44:67:47:97:22:3b + - 44:67:47:97:22:3c + - 44:67:47:97:22:3d + - 44:67:47:97:22:3e +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.2 -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] sbin: load PKGBUILD.src content
by Liu Shaofei 02 Dec '20

02 Dec '20

The contents of PKGBUILD.src is the following example: source=( "http://172.168.131.113:8800/initrd/build-tar/ansible/ansible-2.9.4.tar.gz" ) Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 6f8b95af..498b4872 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -895,6 +895,7 @@ update_pkgver() { sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_safe "PKGBUILD.src" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3699,6 +3700,7 @@ else fi source_buildfile "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_buildfile PKGBUILD.src fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

[PATCH v2 lkp-tests] spec/submit: add job yaml for changing multi-args style
by Bai Jing 02 Dec '20

02 Dec '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 6 ++--- .../unixbench-spec-1-300-pipe.yaml | 25 +++++++++++++++++++ ...unixbench-spec-1-300-whetstone-double.yaml | 8 +++--- .../unixbench-spec-3-300-shell1.yaml | 8 +++--- .../unixbench-spec-3-300-shell8.yaml | 25 +++++++++++++++++++ 5 files changed, 61 insertions(+), 11 deletions(-) create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml index e17322b4..34fb4996 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -2,8 +2,8 @@ suite: unixbench runtime: 300s -nr_task test: - - 1 whetstone-double - - 3 shell1 +nr_task|test: + - 1|pipe + - 3|shell8 unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml new file mode 100644 index 00000000..44be6b10 --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: pipe +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: pipe + nr_task: '1' + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml index 9896bcef..78f79fa7 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '1' @@ -13,7 +13,7 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +21,5 @@ pp: runtime: 300 test: whetstone-double nr_task: '1' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml index 912d10ee..afe5a953 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '3' @@ -13,7 +13,7 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +21,5 @@ pp: runtime: 300 test: shell1 nr_task: '3' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml new file mode 100644 index 00000000..fa58377c --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell8 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell8 + nr_task: '3' + cpufreq_governor: +monitors: -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: 修改名字为如何登录测试机调测任务.md并增加和修改内容
by Zhang Yale 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...03\346\265\213\344\273\273\345\212\241.md" | 69 ++++++++++++++----- 1 file changed, 53 insertions(+), 16 deletions(-) rename doc/manual/how-to-log-in-the-machine.md => "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" (55%) diff --git a/doc/manual/how-to-log-in-the-machine.md "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" similarity index 55% rename from doc/manual/how-to-log-in-the-machine.md rename to "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index fefd718..f4f33a3 100644 --- a/doc/manual/how-to-log-in-the-machine.md +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -1,17 +1,20 @@ -这篇文档将告诉你如何登陆测试环境 +这篇文档将告诉你如何登陆测试环境去调测任务 -# 1. 前提条件 -请先学习: +# 1. 前提条件： +## 请先学习: * [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 * [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 * [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 # 2. 操作方法 -## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： -```shell +## 2.1 如何修改yaml? +### 在测试机运行script之前进入测试机调测任务，可以在job.yaml里加上sshd和sleep字段，测试机在运行脚本之前就sleep, 并免密登录进去，可以手动输入命令或脚本进行调试，以host-inoyaml为例： + +```yaml suite: host-info category: functional + pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -19,6 +22,7 @@ nil end %> + sshd: # sleep at the bottom sleep: 1h @@ -26,19 +30,51 @@ host-info: ``` -## 2.2 以下有2种方式可以登录到测试机： -### 第一种：使用submit -m -c的方式： - 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 - 命令：submit -m -c host-info.yaml - 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： +### 在测试任务运行失败时进入测试机调测任务，可以在job.yaml里加上pub_key和on_fail字段，并在on_fail下加上sshd和sleep字段, 测试任务失败后，测试机保留并免密登录进去，以方便手动调试，以iperf.yaml为例： + +```yaml + suite: iperf + testcase: iperf + category: benchmark + + runtime: 300s + + cluster: cs-localhost + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + if role server: + iperf-server: + + if role client: + iperf: + protocol: + - tcp + - udp + + on_fail: + sshd: + sleep: 1000s +``` +## 如何免密登录测试机? +### submit时加上-m -c，该方法不用查看邮件登录，自动免密登录到测试机： + + 命令：submit -m -c iperf.yaml + 效果如下： ```shell - hi6325@account-vm ~% submit -m -c atomic.yaml - submit atomic.yaml, got job_id=crystal.146528 + hi6325@account-vm ~% submit -m -c iperf.yaml + submit iperf.yaml, got job_id=crystal.146528 query=>{"job_id":["crystal.146528"]} connect to ws://localhost:11310/filter - {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "result_root": "/srv/result/iperf/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" {"job_id": "crystal.146528", "job_state": "boot"} {"job_id": "crystal.146528", "job_state": "download"} "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} @@ -48,9 +84,10 @@ root@vm-2p8g ~# ``` -### 第二种：根据邮件信息使用ssh方式登录测试机： - 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 - 命令： submit host-info.yaml +### 直接submit或者submit只加-m参数，可在收到的邮件里，按照邮件提示手动免密登录到测试机调测： + + 命令：submit iperf.yaml + 任务执行完成后，系统发送邮件内容如下： ```shell -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] send-internet-mail: update start
by Luan Shengde 02 Dec '20

02 Dec '20

attach MAILDIR to the container store mail files that have been sent - MAILDIR/sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 483f01a..d6e4e25 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -12,25 +12,28 @@ names = Set.new %w[ ROBOT_EMAIL_ADDRESS ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT + MAILDIR ] defaults = relevant_defaults(names) defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] ||= '/srv/cci/Maildir/.compass-ci' +defaults['HOST_SERVER'] = ENV['HOSTNAME'] env = docker_env(defaults) -cmd = %W[ +cmd = %w[ docker run --restart=always --name=send-internet-mail -d ] + env + %W[ -p #{defaults['SEND_MAIL_PORT']}:#{defaults['SEND_MAIL_PORT']} + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']} -v /etc/localtime:/etc/localtime:ro -w #{defaults['CCI_SRC']}/container/send-internet-mail - --log-driver json-file send-internet-mail ] -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] send-internet-mail/mail-post: call to store email
by Luan Shengde 02 Dec '20

02 Dec '20

call function store_email to store email to directory sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 0b1f94a..df4d255 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -51,5 +51,11 @@ def send_mail(mail_info) to mail_info['to'] body mail_info['body'] end + mail.deliver! + + return if ENV['SEND_MAIL_PORT'].to_s != '49000' + return if ENV['HOST_SERVER'] != 'z9' + + store_email(mail) end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix upload in container
by Wang Yong 02 Dec '20

02 Dec '20

docker can not use mount operation, so now use curl to upload build-pkg cgz file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef0355..6e435a6ad 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -18,6 +18,8 @@ ## See: https://wiki.archlinux.org/index.php/Makepkg . $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/env.sh +. $LKP_SRC/distro/common check_vars() { @@ -29,6 +31,13 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +mount_pkg_mnt() +{ + [ -n "$LKP_SERVER" ] && { + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + } +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -39,9 +48,7 @@ mount_dest() PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" - [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" - } + is_docker || mount_pkg_mnt } get_pkgfile() @@ -117,10 +124,15 @@ build_source_pkg() } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 - [ "$?" == 0 ] && create_softlink + [ "$?" == 0 ] || exit 1 + create_softlink + + is_docker || exit 0 + upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } check_vars -- 2.23.0

1 0

[PATCH v1 compass-ci 2/2] sched: result root add "." and replace "/" to "-"
by Cao Xueliang 02 Dec '20

02 Dec '20

Before: archlinuxaur-zzookeeper-HEAD After: archlinux-aur-z-zookeeper-HEAD Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index 28ec6ef..da8d8f3 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -51,8 +51,8 @@ class Job private def format_string(original_str) temp = [] of String original_str.each_char do |char| - if "#{char}" =~ /-|\w/ - temp << "#{char}" + if "#{char}" =~ /-|\.|\/|\w/ + temp << "#{char}".gsub("\/", "-") end end -- 2.23.0

2 2

[PATCH compass-ci] scheduler: fix: 'lab' not 'LAB' in scheduler environment
by Ren Wen 02 Dec '20

02 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..418e7ff 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -10,7 +10,7 @@ JOB_ES_PORT = 9200 JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" -LAB = (ENV.has_key?("LAB") ? ENV["LAB"] : "nolab") +LAB = (ENV.has_key?("lab") ? ENV["lab"] : "nolab") SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit: add job yaml for changing multi-args style
by Bai Jing 02 Dec '20

02 Dec '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 6 +++--- .../unixbench-spec-1-300-whetstone-double.yaml | 11 +++++++---- .../unixbench-spec/unixbench-spec-3-300-shell1.yaml | 11 +++++++---- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml index e17322b4..851f0473 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -2,8 +2,8 @@ suite: unixbench runtime: 300s -nr_task test: - - 1 whetstone-double - - 3 shell1 +nr_task|test: + - 1|whetstone-double + - 3|shell1 unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml index 9896bcef..6a29541f 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '1' @@ -13,7 +13,10 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +24,5 @@ pp: runtime: 300 test: whetstone-double nr_task: '1' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml index 912d10ee..3e49255e 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '3' @@ -13,7 +13,10 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +24,5 @@ pp: runtime: 300 test: shell1 nr_task: '3' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: -- 2.23.0

1 0

[PATCH compass-ci] define_files.md: adjust markdown document format
by Xu Xijian 02 Dec '20

02 Dec '20

In markdown documents, text for bash should be contained in block as below: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/define_files.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/job/define_files.md b/doc/job/define_files.md index 2c3457d..e418b4f 100644 --- a/doc/job/define_files.md +++ b/doc/job/define_files.md @@ -2,15 +2,19 @@ Meaning: - We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: - distro/depends/$testcase +```SHELL + distro/depends/$testcas distro/depends/$testcase-dev pkg/$testcase/PKGBUILD stats/$testcase tests/$testcase jobs/$testcase.yaml +``` - When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. - The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. Usage example: +```SHELL - % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 - % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml +``` -- 2.23.0

1 0