- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/2] lib/json_logger.rb: refactor log.rb
by Wu Zhende 24 Nov '20

24 Nov '20

1. change log.rb to json_logger.rb 2. change Log to JSONLogger 3. change datetime to time 4. don't output empty values 5. msg use hash or string(originally json or string) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/{log.rb => json_logger.rb} | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) rename lib/{log.rb => json_logger.rb} (65%) diff --git a/lib/log.rb b/lib/json_logger.rb similarity index 65% rename from lib/log.rb rename to lib/json_logger.rb index aad00ca..6d1173e 100644 --- a/lib/log.rb +++ b/lib/json_logger.rb @@ -6,7 +6,7 @@ require 'logger' require 'json' # print logs in JSON format -class Log < Logger +class JSONLogger < Logger LEVEL_INFO = { 'DEBUG' => 0, 'INFO' => 1, @@ -18,21 +18,19 @@ class Log < Logger FORMATTER = proc { |severity, datetime, progname, msg| level_num = LEVEL_INFO[severity] - begin - message = JSON.parse(msg) - rescue JSON::ParserError - message = { 'message' => msg } - end - h = { + logger_hash = { 'level' => severity.to_s, 'level_num' => level_num, - 'datetime' => datetime, - 'progname' => progname, - 'message' => '' + 'time' => datetime } - h.merge!(message) - h.merge!({ 'caller' => caller }) if level_num >= 2 - h.to_json + + logger_hash['progname'] = progname if progname + logger_hash['caller'] = caller if level_num >= 2 + + msg = { 'message' => msg } unless msg.is_a?(Hash) + logger_hash.merge!(msg) + + logger_hash.to_json } def initialize(logdev = STDOUT, formatter = FORMATTER) -- 2.23.0

1 0

[PATCH v6 compass-ci 2/2] container/result-webdav: add symbolic link after uploading file
by Cui Yili 24 Nov '20

24 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..c880522 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -13,8 +13,10 @@ http { server_name "result-webdav"; server_tokens off; client_max_body_size 100m; + access_log /tmp/access.log; + error_log /tmp/error.log; - location / { + location /result { allow all; root /srv/; @@ -39,6 +41,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + upload_file = string.match(uri, ".+/([^/]*%.%w+)$") + second_dir = string.match(uri, "/.+/(.*)/.+") + + if (second_dir == "pkg") + then + link_name = "latest.cgz" + else + link_name = string.match(upload_file, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) + } + } } } -- 2.23.0

1 0

[PATCH v5 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 24 Nov '20

24 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..1085492 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + second_dir = string.match(uri, "/.+/(.*)/.+") + + if (second_dir == "pkg") + then + link_name = "latest.cgz" + else + link_name = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..link_name) + } + } } } -- 2.23.0

3 2

[PATCH lkp-tests] makepkg.conf: mv -Wl from LDFLAGS to C(CXX)FLAGS
by Xu Xijian 24 Nov '20

24 Nov '20

-Wl parameter should be used by CFLAGS and CXXFLAGS instead of LDFLAGS. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- etc/makepkg.conf | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 093789f5..1fffdc2d 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -36,9 +36,8 @@ VCSCLIENTS=('bzr::bzr' # -march (or -mcpu) builds exclusively for an architecture # -mtune optimizes for an architecture, but builds for whole processor family CPPFLAGS="-D_FORTIFY_SOURCE=2" -CFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt" -CXXFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt" -LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" +CFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" +CXXFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" #-- Make Flags: change this for DistCC/SMP systems #MAKEFLAGS="-j2" #-- Debugging flags -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 24 Nov '20

24 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h2 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} Because "@job2=hash" will merge the h2 to @job after "each_job". We need to merge the h2 to @job before "each_job". So do the merge advance and set @job2={} Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index bc6c91ef..2062ba58 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 24 Nov '20

24 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 2e94316c..bc6c91ef 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: fix syntax errors caused by replacement
by Lin Jiaxin 24 Nov '20

24 Nov '20

error: //PKGBUILD: line 13: syntax error near unexpected token `)' //PKGBUILD: line 13: ` "$_pkgname.desktop")' The original logic only adapts to the situation where source has only one value. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index c5d55eda..65c83e22 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -90,6 +90,11 @@ create_softlink() ln -sf "${bm_name}" "${soft_path}/latest.cgz" } +replace_source() +{ + sed -i "s|^source=[^)]*|$1|g" "$2" +} + build_source_pkg() { local repo_dir="" @@ -103,11 +108,9 @@ build_source_pkg() repo_dir="${repo_dir%.git*}::" fi - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" - sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD - [ -n "$PKGBUILD_TAG" ] && { - sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" - } + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + replace_source "${upstream_source}" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job2sh.rb: the job value obtaining logic is modified
by Hu Xuejiao 24 Nov '20

24 Nov '20

[why] the original logic is too round Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 14bf7159..7f9cf562 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -287,9 +287,7 @@ class Job2sh < Job @setups = available_programs(:setup) @programs = available_programs(:workload_elements) - if !job - job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job - end + job ||= (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job @cur_func = :extract_stats @@ -317,7 +315,7 @@ class Job2sh < Job @setups = available_programs(:setup) @programs = available_programs(:workload_elements) - if !job + unless job job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job job.delete_if { |key, val| parse_one([], key, val, :PASS_EXPORT_ENV) } end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/wrapper support user define file to parse result
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] User need debug their script which haven't commit to lkp-tests/stats [How] lkp-tests/stats/wrapper calls the program-script in result_root if lkp-tests/stats/ have no program-script [Dependence] Write job['define_files'] to file in result_root, job can query from ES DB Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/wrapper | 3 +++ 1 file changed, 3 insertions(+) diff --git a/stats/wrapper b/stats/wrapper index a44e9f31..d13249d2 100755 --- a/stats/wrapper +++ b/stats/wrapper @@ -140,6 +140,9 @@ check_empty_output check_binary_output tmpfile=$(mktemp -p /tmp -t lkp-stats.XXXXXXXX) + +[[ -f "$RESULT_ROOT/stats/$program" ]] && program_dir="$RESULT_ROOT/stats" + if [[ -f $log ]]; then $program_dir/$program $log < $log > $tmpfile || { log_error "$program_dir/$program $log < $log exit code $?, check $tmpfile"; exit 1; } else -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] doc/job/: add define_files.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the define_files field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/define_files.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/define_files.md diff --git a/doc/job/define_files.md b/doc/job/define_files.md new file mode 100644 index 0000000..2c3457d --- /dev/null +++ b/doc/job/define_files.md @@ -0,0 +1,16 @@ +# define_files + +Meaning: +- We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: + distro/depends/$testcase + distro/depends/$testcase-dev + pkg/$testcase/PKGBUILD + stats/$testcase + tests/$testcase + jobs/$testcase.yaml +- When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. +- The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. + +Usage example: +- % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 +- % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml -- 2.23.0

1 0