November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sched: save the upload dirs in redis
by Cao Xueliang 23 Nov '20

23 Nov '20

Before: Use access_key_file to limit job upload, need to create the file in upload dirs, that's not flexible enough. After: Set $job_id => "$dir1,$dir2,..." in "id2upload_dirs" hash of the redis. The dir of job upload must be equal or deeper than one of "$dir1,$dir2,...". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 13 ++++--------- src/scheduler/close_job.cr | 6 ------ src/scheduler/find_job_boot.cr | 10 +++++----- 3 files changed, 9 insertions(+), 20 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..cb187a4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -83,8 +83,7 @@ class Job pp_params submit_date result_root - access_key - access_key_file + upload_dirs lkp_initrd_user user_lkp_src kernel_uri @@ -206,15 +205,11 @@ class Job def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") - - # access_key has information based on "result_root" - # so when set result_root, we need redo set_ to update it - set_access_key() + set_upload_dirs() end - private def set_access_key - self["access_key"] = "#{Random::Secure.hex(10)}-#{id}" - self["access_key_file"] = File.join("/srv/", "#{result_root}", ".#{access_key}") + def set_upload_dirs + self["upload_dirs"] = "#{result_root}" end private def set_result_service diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d071d69..b5aaa63 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -5,8 +5,6 @@ class Sched def close_job(job_id : String) job = @redis.get_job(job_id) - delete_access_key_file(job) if job - response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -24,8 +22,4 @@ class Sched return %({"job_id": "#{job_id}", "job_state": "complete"}) end - - def delete_access_key_file(job : Job) - File.delete(job.access_key_file) if File.exists?(job.access_key_file) - end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..968b9ed 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -128,10 +128,6 @@ class Sched return response end - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" response += job.initrds_uri @@ -142,8 +138,12 @@ class Sched return response end + def set_id2upload_dirs(job) + @redis.hash_set("sched/id2upload_dirs",job.id, job.upload_dirs) + end + def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job + set_id2upload_dirs(job) if job case boot_type when "ipxe" -- 2.23.0

1 1

[PATCH v3 compass-ci] sbin/es-jobs-mapping.sh: Support more fields in "jobs" index
by Zhang Yuhang 23 Nov '20

23 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..8e19662 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,9 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ + -d '{"index.mapping.total_fields.limit": 10000}' fi fi -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: set the upload dirs to redis
by Cao Xueliang 23 Nov '20

23 Nov '20

Before: Use access_key_file to limit job upload, need to create the file in upload dirs, that's not flexible enough. After: Set $job_id => "$dir1,$dir2,..." in "id2upload_dirs" hash of the redis. The dir of job upload must be equal or deeper than one of "$dir1,$dir2,...". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 13 ++++--------- src/scheduler/close_job.cr | 6 ------ src/scheduler/find_job_boot.cr | 10 +++++----- 3 files changed, 9 insertions(+), 20 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..cb187a4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -83,8 +83,7 @@ class Job pp_params submit_date result_root - access_key - access_key_file + upload_dirs lkp_initrd_user user_lkp_src kernel_uri @@ -206,15 +205,11 @@ class Job def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") - - # access_key has information based on "result_root" - # so when set result_root, we need redo set_ to update it - set_access_key() + set_upload_dirs() end - private def set_access_key - self["access_key"] = "#{Random::Secure.hex(10)}-#{id}" - self["access_key_file"] = File.join("/srv/", "#{result_root}", ".#{access_key}") + def set_upload_dirs + self["upload_dirs"] = "#{result_root}" end private def set_result_service diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d071d69..b5aaa63 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -5,8 +5,6 @@ class Sched def close_job(job_id : String) job = @redis.get_job(job_id) - delete_access_key_file(job) if job - response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -24,8 +22,4 @@ class Sched return %({"job_id": "#{job_id}", "job_state": "complete"}) end - - def delete_access_key_file(job : Job) - File.delete(job.access_key_file) if File.exists?(job.access_key_file) - end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..968b9ed 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -128,10 +128,6 @@ class Sched return response end - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" response += job.initrds_uri @@ -142,8 +138,12 @@ class Sched return response end + def set_id2upload_dirs(job) + @redis.hash_set("sched/id2upload_dirs",job.id, job.upload_dirs) + end + def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job + set_id2upload_dirs(job) if job case boot_type when "ipxe" -- 2.23.0

1 0

[PATCH v3 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..c9199ed 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

2 1

[PATCH compass-ci 2/2] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..0cb786c 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -13,8 +13,10 @@ http { server_name "result-webdav"; server_tokens off; client_max_body_size 100m; + access_log /tmp/access.log; + error_log /tmp/error.log; - location / { + location /result { allow all; root /srv/; @@ -39,6 +41,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

2 1

[PATCH v2 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..34df004 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

1 0

[PATCH compass-ci 1/2] result-webdav/start: add share dir to upload file
by Cui Yili 23 Nov '20

23 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index e0b6676..01141c3 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -14,6 +14,7 @@ cmd=( -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/result/:/srv/result/ + -v /srv/initrd/:/srv/initrd/ result-webdav ) -- 2.23.0

1 0

[PATCH v2 compass-ci] add api /compare_template
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] user need to compare the specific metrics [how] add new api: /compare_template that can be access like: curl -X POST 'localhost:32767/compare_template' -H 'Content-Type: Application/json' -d '{ "compare_metrics": ["fio.write_iops", "fio.read_iops"], "filter": {"suite": ["fio-basic"], "os_arch": ["aarch64", "x86"]}, "compare_dimensions": [ {"os": "debian", "os_version": "sid"}, {"os": "openeuler", "os_version": 20.03} ], "x_params": ["bs", "test_size"], "title": "Hackbench Performance Testing", "unit": "KB/s"}' Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 32 +++++++++++++++++++++++++++++++ src/lib/web_backend.rb | 18 +++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 7db82ac..d68e9f2 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -71,3 +71,35 @@ end get '/get_repos' do get_repos(params) end + +# POST /compare_template +# - header: "Content-Type: Application/json" +# - body: json +# eg: +# { +# "compare_metrics": ["fio.write_iops", "fio.read_iops"], +# "filter":{ +# "suite": ["fio-basic"], +# "os_arch": ["aarch64", "x86"] +# }, +# "compare_dimensions": [ +# { +# "os": "debian", +# "os_version": "sid" +# }, +# { +# "os": "openeuler", +# "os_version: 20.03 +# } +# ], +# "x_params": ["bs", "test_size"], +# "title": "Hackbench Performance Testing", +# "unit": "KB/s" +# } +# +# Response: +# - compare result +post '/compare_template' do + request.body.rewind # in case someone already read it + compare_template(request.body.read) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..23716cb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -413,3 +414,20 @@ def get_repos(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def compare_template(data) + begin + body = template_body(JSON.parse(data)) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'compare error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def template_body(request_body) + groups_matrices = create_groups_matrices(request_body) + compare_results = compare_metrics_values(groups_matrices) + formatter = FormatEchartData.new(compare_results, request_body) + formatter.format_for_echart.to_json +end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix syntax errors caused by replacement
by Lin Jiaxin 23 Nov '20

23 Nov '20

error: //PKGBUILD: line 13: syntax error near unexpected token `)' //PKGBUILD: line 13: ` "$_pkgname.desktop")' The original logic only adapts to the situation where source has only one value. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index c5d55eda..6ad825d7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -103,10 +103,10 @@ build_source_pkg() repo_dir="${repo_dir%.git*}::" fi - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" - sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + sed -i "s|^source=[^)]*|${upstream_source}|g" PKGBUILD [ -n "$PKGBUILD_TAG" ] && { - sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" + sed -i "s|^source=[^)]*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" } } -- 2.23.0

2 1

[PATCH lkp-tests] stats/wrapper support user define file to parse result
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] User need debug their script which haven't commit to lkp-tests/stats [How] lkp-tests/stats/wrapper calls the program-script in result_root if lkp-tests/stats/ have no program-script [Dependence] Write job['define_files'] to file in result_root, job can query from ES DB Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/wrapper | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/stats/wrapper b/stats/wrapper index a44e9f31..03db5975 100755 --- a/stats/wrapper +++ b/stats/wrapper @@ -140,6 +140,13 @@ check_empty_output check_binary_output tmpfile=$(mktemp -p /tmp -t lkp-stats.XXXXXXXX) + +if [[ ! -f $program_dir/$program ]]; then + if [[ -f $RESULT_ROOT/"stats"/$program ]]; then + program_dir=$RESULT_ROOT/"stats" + fi +fi + if [[ -f $log ]]; then $program_dir/$program $log < $log > $tmpfile || { log_error "$program_dir/$program $log < $log exit code $?, check $tmpfile"; exit 1; } else -- 2.23.0

2 2