- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/minio: add minio container
by Cui Yili 16 Nov '20

16 Nov '20

Add minio container to store files. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/minio/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/minio/build diff --git a/container/minio/build b/container/minio/build new file mode 100755 index 0000000..d1ccd06 --- /dev/null +++ b/container/minio/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker pull jessestuart/minio -- 2.23.0

2 1

[PATCH v4 compass-ci] /lib: keep suite result in top for compare output
by Lu Kaiyi 16 Nov '20

16 Nov '20

[why] Currently, the compare output result default order by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. [example] command: compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 15 +++++++++++---- 3 files changed, 49 insertions(+), 23 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..813e08c 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suites_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suites_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suites_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suites_list.concat(suites) end - matrices_list + + return matrices_list, suites_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..bd1e646 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suites_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suites_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suites_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suites_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suites_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suites_list, matrixes_titles, success, theme) return '' if values.empty? + suites_list.uniq! result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suites_list, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suites_list, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suites_list.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..479d280 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,20 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end groups.delete(group_key) if value.size < 2 + suites_list << suite_list unless value.size < 2 end - groups + + return groups, suites_list end # input: @@ -139,7 +146,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

3 4

[PATCH v4 compass-ci 4/4] container/git-mirrir: add Docker env GIT_ASKPASS
by Li Yuanchao 16 Nov '20

16 Nov '20

[why] some repositories need username and password, but we don't know. And the problem is, it will block there as we have no username and password. Add GIT_ASKPASS=/kill-parent.sh, will kill the block one, and go to next repository. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/Dockerfile | 3 +++ container/git-mirror/kill-parent.sh | 3 +++ container/git-mirror/start | 1 + 3 files changed, 7 insertions(+) create mode 100755 container/git-mirror/kill-parent.sh diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 390bbcc..1180a7b 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -2,6 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM alpine:3.8 +MAINTAINER Li Yuanchao <lyc163mail(a)163.com> + +ADD kill-parent.sh / RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories diff --git a/container/git-mirror/kill-parent.sh b/container/git-mirror/kill-parent.sh new file mode 100755 index 0000000..6418592 --- /dev/null +++ b/container/git-mirror/kill-parent.sh @@ -0,0 +1,3 @@ +#!/bin/sh + +kill $PPID diff --git a/container/git-mirror/start b/container/git-mirror/start index dd4fd63..ee79578 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -17,6 +17,7 @@ cmd = %W[ -d -e CCI_SRC=#{DOCKER_CCI} -e REPO_SRC=#{DOCKER_REPO} + -e GIT_ASKPASS=/kill-parent.sh -v #{ENV['CCI_SRC']}:#{DOCKER_CCI} -v /etc/localtime:/etc/localtime:ro -v #{ENV['REPO_SRC']}:#{DOCKER_REPO} -- 2.23.0

1 0

[PATCH v4 compass-ci 3/4] git_mirror: use "gitclone.com" to make "git clone" faster
by Li Yuanchao 16 Nov '20

16 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 8f0497e..9648063 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -31,12 +31,18 @@ class GitMirror @feedback_queue.push(@feedback_info) end - def git_clone(url, mirror_dir) - ret = false - url = Array(url)[0] + def get_url(url) if url.include?('gitee.com/') && File.exist?("/srv/git/#{url.delete_prefix('https://')}") url = "/srv/git/#{url.delete_prefix('https://')}" + elsif url.include?('://github.com') + url = "https://gitclone.com/#{url.split('://')[1]}" end + return url + end + + def git_clone(url, mirror_dir) + ret = false + url = get_url(Array(url)[0]) 10.times do stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) ret = !stderr.include?('fatal') -- 2.23.0

1 0

[PATCH v4 compass-ci 2/4] git_mirror: fix that worker thread die after system(git clone)
by Li Yuanchao 16 Nov '20

16 Nov '20

[why] As Signal.trap(:SIGCHLD, 'SIG_IGN') has been set, the init process will handle the return status of child process, so ret = system(git clone xxx) can't get return value, which cause worker thread die. Use %x() instead of system(), and use stderr to check if 'git clone' succeed. So return value is not needed. [errmsg] in `system': No child process - Another thread waited the process started by system(). (Errno::ECHILD) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 55ec67d..8f0497e 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -38,7 +38,8 @@ class GitMirror url = "/srv/git/#{url.delete_prefix('https://')}" end 10.times do - ret = system("git clone --mirror #{url} #{mirror_dir}") + stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) + ret = !stderr.include?('fatal') break if ret end FileUtils.rm_r(mirror_dir) unless ret -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add optimization parameters to job.yaml
by Zhang Yu 16 Nov '20

16 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- jobs/sysbench-mysql.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml index 83795a18..12c83087 100644 --- a/jobs/sysbench-mysql.yaml +++ b/jobs/sysbench-mysql.yaml @@ -2,6 +2,8 @@ suite: sysbench-mysql category: benchmark cluster: cs-lkp-hsw-ep5 +kernel_version: 4.19.90-mysql +kernel_custom_params: "sched_steal_node_limit=8" if role server: mysql: -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: support more testbox formats
by Li Ping 16 Nov '20

16 Nov '20

If user wants to reproduce the result directly in the result directory. [problem] /srv/result/iperf/vm-2p8g/2020-11-09/z9.147520% submit job.yaml output: /home/liping/lkp-tests/lib/job.rb:344:in `get_hosts_file': hosts file not exist: vm-2p8g.taishan200-2280-2s64p-256g, maybe need check testbox field (ArgumentError) --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..6dd93f01 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -337,7 +337,7 @@ class Job lab_hosts_file = get_lab_hosts_file return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split('--')[0] + hosts_file_name = @job['tbox_group'].split(/(\.|--)/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" return hosts_file if File.file?(hosts_file) -- 2.23.0

2 2

[PATCH v4 compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 16 Nov '20

16 Nov '20

when execute 3-code/dev-env, error message as follows: ./dev-env: line 45: ./os/openEuler: No such file or directory no file ./os/openEuler is normal, will not affect deploy CCI, so fix it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 77027d9..9fcc1b4 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -42,4 +42,4 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +. $(dirname ${BASH_SOURCE[0]})/os/${ID} 2> /dev/null -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 15 Nov '20

15 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..1faa1e9f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,6 +37,23 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +check_pkgfile() +{ + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + + get_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 @@ -69,6 +86,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && check_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 1

[PATCH compass-ci] lib/es-query: fix NoMethodError
by Lu Weitao 14 Nov '20

14 Nov '20

[Why] Traceback (most recent call last): 4: from /c/compass-ci/sbin/compare:70:in `<main>' 3: from /c/compass-ci/lib/compare.rb:97:in `compare_by_template' 2: from /c/compass-ci/lib/compare.rb:128:in `create_groups_matrices' 1: from /c/compass-ci/lib/es_query.rb:21:in `multi_field_query' /c/compass-ci/lib/es_query.rb:54:in `build_mutli_field_subquery_body': undefined method `each' for nil:NilClass (NoMethodError) [Root cause] when (sbin/compare) call es_query, may pass empty filter as input parameter [How] exit when the filter is empty Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 037213d..c027aa9 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,6 +18,10 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) def multi_field_query(items, size: 10_000) + unless items + warn 'empty filter!' + exit + end query_fields = build_mutli_field_subquery_body items query = { query: { -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/makepkg: download submodules code from LKP server
by Liu Shaofei 14 Nov '20

14 Nov '20

Some repositories have submodules. So when build packages, we need download submodules from LKP server, avoid the waste of time in downloading files from the remote end. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index fde82212..c8b2ed00 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -623,6 +623,11 @@ extract_git() { cd_safe "${dir##*/}" + # modify submodules address to local server address. + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> ~/.gitconfig + fi + local ref=origin/HEAD if [[ -n $fragment ]]; then case ${fragment%%=*} in -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: change the result root dir
by Cao Xueliang 14 Nov '20

14 Nov '20

The new result root dir: /srv/result/$suite/$submit_date/$tbox_group/$rootfs/$pp_params/$job_id Example: rootfs: openeuler-20.09-aarch64 ($os-$os_version-$os_aarch) pp_params: socket-process-50 (use "-" connect pp's value up to 40 size) Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 18 +++++++++-- src/scheduler/pp_params.cr | 61 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 src/scheduler/pp_params.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index be0fdbb..4bef329 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -8,6 +8,7 @@ require "any_merge" require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" +require "scheduler/pp_params.cr" struct JSON::Any def []=(key : String, value : String) @@ -78,6 +79,9 @@ class Job initrd_pkg initrd_deps initrds_uri + rootfs + pp_params + submit_date result_root access_key access_key_file @@ -135,6 +139,9 @@ class Job append_init_field() set_user_lkp_src() set_os_dir() + set_submit_date() + set_pp_params() + set_rootfs() set_result_root() set_result_service() set_os_mount() @@ -188,10 +195,17 @@ class Job self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end + private def set_submit_date + self["submit_date"] = Time.local.to_s("%F") + end + + private def set_rootfs + self["rootfs"] = "#{os}-#{os_version}-#{os_arch}" + end + def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group - date = Time.local.to_s("%F") - self["result_root"] = "/result/#{suite}/#{tbox_group}/#{date}/#{id}" + self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") # access_key has information based on "result_root" # so when set result_root, we need redo set_ to update it diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr new file mode 100644 index 0000000..28ec6ef --- /dev/null +++ b/src/scheduler/pp_params.cr @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + private def set_pp_params + temp_hash = Hash(String, JSON::Any).new + + if @hash["pp"] + flat_hash(@hash["pp"].as_h, temp_hash) + end + + self["pp_params"] = sort_keys_return_values(temp_hash) + end + + private def flat_hash(old_hash, new_hash) + old_hash.each do |key1, value1| + if value1.as_h? + next if value1.as_h.empty? + + temp_hash = Hash(String, JSON::Any).new + value1.as_h.each do |key2, value2| + temp_hash.merge!({"#{key1}-#{key2}" => value2}) + end + + flat_hash(temp_hash, new_hash) + else + new_hash.merge!({key1 => value1}) + end + end + end + + private def sort_keys_return_values(hash) + values = [] of String + items_size = 0 + + hash.keys.sort.each do |key| + value = format_string(hash[key].to_s) + next if 0 == value.size + + if 40 > value.size + items_size + values.size + values << value + items_size += value.size + else + break + end + end + + return values.join("-") + end + + private def format_string(original_str) + temp = [] of String + original_str.each_char do |char| + if "#{char}" =~ /-|\w/ + temp << "#{char}" + end + end + + return temp.join() + end +end -- 2.23.0

1 0

[PATCH compass-ci] sched: change the result root dir
by Cao Xueliang 14 Nov '20

14 Nov '20

The new result root dir: /srv/result/$suite/$submit_date/$tbox_group/$rootfs/$pp_params/$job_id Example: rootfs: openeuler-20.09-aarch64 ($os-$os_version-$os_aarch) pp_params: socket-process-50 (use "-" connect pp's value up to 40 size) Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 22 ++++++++++++-- src/scheduler/pp_params.cr | 61 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+), 2 deletions(-) create mode 100644 src/scheduler/pp_params.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index be0fdbb..f723100 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -8,6 +8,7 @@ require "any_merge" require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" +require "scheduler/pp_params.cr" struct JSON::Any def []=(key : String, value : String) @@ -78,6 +79,9 @@ class Job initrd_pkg initrd_deps initrds_uri + rootfs + pp_params + submit_date result_root access_key access_key_file @@ -135,6 +139,9 @@ class Job append_init_field() set_user_lkp_src() set_os_dir() + set_submit_date() + set_pp_params() + set_rootfs() set_result_root() set_result_service() set_os_mount() @@ -188,10 +195,21 @@ class Job self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end + private def set_submit_date + self["submit_date"] = Time.local.to_s("%F") + end + + private def set_rootfs + self["rootfs"] = "#{os}-#{os_version}-#{os_arch}" + end + def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group - date = Time.local.to_s("%F") - self["result_root"] = "/result/#{suite}/#{tbox_group}/#{date}/#{id}" + if pp_params != "" + self["result_root"] = "/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}/#{pp_params}/#{id}" + else + self["result_root"] = "/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}/#{id}" + end # access_key has information based on "result_root" # so when set result_root, we need redo set_ to update it diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr new file mode 100644 index 0000000..28ec6ef --- /dev/null +++ b/src/scheduler/pp_params.cr @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + private def set_pp_params + temp_hash = Hash(String, JSON::Any).new + + if @hash["pp"] + flat_hash(@hash["pp"].as_h, temp_hash) + end + + self["pp_params"] = sort_keys_return_values(temp_hash) + end + + private def flat_hash(old_hash, new_hash) + old_hash.each do |key1, value1| + if value1.as_h? + next if value1.as_h.empty? + + temp_hash = Hash(String, JSON::Any).new + value1.as_h.each do |key2, value2| + temp_hash.merge!({"#{key1}-#{key2}" => value2}) + end + + flat_hash(temp_hash, new_hash) + else + new_hash.merge!({key1 => value1}) + end + end + end + + private def sort_keys_return_values(hash) + values = [] of String + items_size = 0 + + hash.keys.sort.each do |key| + value = format_string(hash[key].to_s) + next if 0 == value.size + + if 40 > value.size + items_size + values.size + values << value + items_size += value.size + else + break + end + end + + return values.join("-") + end + + private def format_string(original_str) + temp = [] of String + original_str.each_char do |char| + if "#{char}" =~ /-|\w/ + temp << "#{char}" + end + end + + return temp.join() + end +end -- 2.23.0

2 2

[PATCH compass-ci] container/kibana: fix build error
by Wu Zhende 14 Nov '20

14 Nov '20

Sending build context to Docker daemon 5.12kB Step 1/4 : ARG BASE_IMAGE Step 2/4 : FROM BASE_IMAGE invalid reference format: repository name must be lowercase Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 6e0dba0..970eb5a 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -3,7 +3,7 @@ ARG BASE_IMAGE -FROM BASE_IMAGE +FROM ${BASE_IMAGE} # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/3] iso2rootfs: add some configurations after generated rootfs
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] The initial rootfs have the follow two problems: - no dns resolver; - selinux is opened; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4f7843d7f348..05274164f8c1 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -338,6 +338,27 @@ run_qcow2rootfs() ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" } + +############ config rootfs ############ +config_dns_resolver() +{ + cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" + nameserver 114.114.114.114 + nameserver 8.8.8.8 + EOF +} + +disable_selinux() +{ + sed -i 's/^SELINUX=.*/SELINUX=disabled/g' "${ROOTFS_DES_DIR}/etc/selinux/config" +} + +config_rootfs() +{ + config_dns_resolver + disable_selinux +} + ############ test rootfs ############ get_qemu_efi_fd() { @@ -499,6 +520,8 @@ main() config_qcow2rootfs run_qcow2rootfs + config_rootfs + test_rootfs submit_test_job_yaml -- 2.23.0

2 2

[PATCH lkp-tests] tests/cci-makepkg: update gcc for centos7 when use initramfs os
by Lin Jiaxin 13 Nov '20

13 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 67d055a4..bdb2128d 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -88,13 +88,22 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +centos_update_gcc() +{ + if $(grep -sqF "CentOS Linux release 7" /etc/centos-release ); then + [[ "$os_mount" = "initramfs" ]] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable + fi +} + [ -n "$PKGBUILD_TAG" ] && get_pkgfile distro_install_depends lkp-dev -[ $os = "centos" ] && -grep -sqF "CentOS Linux release 7" /etc/centos-release && -source /opt/rh/devtoolset-8/enable +[ $os = "centos" ] && centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

1 0

[PATCH lkp-tests 2/2] depends: add mysql-dev
by Lin Jiaxin 13 Nov '20

13 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/mysql-dev | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 distro/depends/mysql-dev diff --git a/distro/depends/mysql-dev b/distro/depends/mysql-dev new file mode 100644 index 00000000..1434c26c --- /dev/null +++ b/distro/depends/mysql-dev @@ -0,0 +1,5 @@ +gcc +cmake +libssl-dev +libncurses5-dev +libtirpc-dev -- 2.23.0

1 0

[PATCH lkp-tests 1/2] pkg: add mysql
by Lin Jiaxin 13 Nov '20

13 Nov '20

sysbench-mysql need to test with mysql8.0.17 Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...00-mysql-add-fstack-protector-strong.patch | 20 +++++ pkg/mysql/PKGBUILD | 50 ++++++++++++ pkg/mysql/my.cnf | 80 +++++++++++++++++++ 3 files changed, 150 insertions(+) create mode 100644 pkg/mysql/0000-mysql-add-fstack-protector-strong.patch create mode 100644 pkg/mysql/PKGBUILD create mode 100644 pkg/mysql/my.cnf diff --git a/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch new file mode 100644 index 00000000..23c07c87 --- /dev/null +++ b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch @@ -0,0 +1,20 @@ +diff --git a/cmake/build_configurations/compiler_options.cmake b/cmake/build_configurations/compiler_options.cmake +index fecf2a2..f3e012f 100644 +--- a/cmake/build_configurations/compiler_options.cmake ++++ b/cmake/build_configurations/compiler_options.cmake +@@ -43,6 +43,7 @@ IF(UNIX) + # Default GCC flags + IF(CMAKE_COMPILER_IS_GNUCC) + SET(COMMON_C_FLAGS "-fno-omit-frame-pointer") ++ SET(COMMON_C_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_C_FLAGS "-fno-inline ") +@@ -57,6 +58,7 @@ IF(UNIX) + ENDIF() + IF(CMAKE_COMPILER_IS_GNUCXX) + SET(COMMON_CXX_FLAGS "-std=c++14 -fno-omit-frame-pointer") ++ SET(COMMON_CXX_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_CXX_FLAGS "-fno-inline ") diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD new file mode 100644 index 00000000..f293f22f --- /dev/null +++ b/pkg/mysql/PKGBUILD @@ -0,0 +1,50 @@ +pkgname=mysql +pkgver=8.0.17 +pkgrel=3 +arch=('aarch64' 'x86_64') +url="http://www.mysql.com" +license=('GPLv2') +source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" + "0000-mysql-add-fstack-protector-strong.patch" + "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "my.cnf") +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'f1af6a77287e2456e867ecf66166679f') + +prepare() +{ + cd $srcdir/$pkgname-$pkgver + patch -p1 < ../0000-mysql-add-fstack-protector-strong.patch +} + +build() +{ + cd $srcdir/$pkgname-$pkgver + + CMAKE=cmake + grep -sqF "CentOS Linux release 7" /etc/centos-release && CMAKE=cmake3 + + $CMAKE . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc \ + -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 \ + -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_MYISAM_STORAGE_ENGINE=1 \ + -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DDEFAULT_CHARSET=utf8mb4 -DDEFAULT_COLLATION=utf8mb4_general_ci \ + -DWITH_EMBEDDED_SERVER=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=./boost -DFORCE_INSOURCE_BUILD=1 + make -j8 + + cd $srcdir + tar -zxvf getcpu.tgz + + cd getcpu || exit + make +} + +package() +{ + mkdir -p $pkgdir/getcpu + cp getcpu/getcpu-hi.ko $pkgdir/getcpu/ + + mkdir -p $pkgdir/etc + cp my.cnf $pkgdir/etc/ + + cd $srcdir/$pkgname-$pkgver + make DESTDIR=$pkgdir install +} diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf new file mode 100644 index 00000000..8d0b9c05 --- /dev/null +++ b/pkg/mysql/my.cnf @@ -0,0 +1,80 @@ +[mysqld_safe] +log-error=/data/mysql/log/mysql.log +pid-file=/data/mysql/run/mysqld.pid +[mysqldump] +quick +[mysql] +[client] +default-character-set=utf8mb4 +socket=/data/mysql/run/mysql.sock + +[mysqld] +socket=/data/mysql/run/mysql.sock +tmpdir=/data/mysql/tmp +datadir=/data/mysql/data +default_authentication_plugin=mysql_native_password +port=3309 +user=mysql +memlock=on +bind-address=0.0.0.0 +skip-grant-tables +#skip-name-resolve + +lc-messages=en_US +lc-messages-dir=/usr/local/mysql/share + +innodb_page_size=64k +max_connections=4000 +back_log=2048 +performance_schema=OFF +max_prepared_stmt_count=128000 +#transaction_isolation=READ_COMMITTED + +#file +innodb_file_per_table +innodb_log_file_size=2048M +innodb_log_files_in_group=32 +innodb_open_files=10000 +table_open_cache_instances=64 + +#buffers +innodb_buffer_pool_size=128G +innodb_buffer_pool_instances=64 +innodb_log_buffer_size=2048M + +#tune +sync_binlog=0 +innodb_flush_log_at_trx_commit=0 +innodb_use_native_aio=1 +innodb_spin_wait_delay=160 +innodb_sync_spin_loops=20 +innodb_flush_method=O_DIRECT +innodb_io_capacity=30000 +innodb_io_capacity_max=40000 +innodb_lru_scan_depth=9000 +innodb_page_cleaners=1 +innodb_spin_wait_pause_multiplier=25 + +#perf special +innodb_flush_neighbors=0 +innodb_write_io_threads=1 +innodb_read_io_threads=1 +innodb_purge_threads=1 + +sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,STRICT_ALL_TABLES + +#skop_log_bin +#log-bin=mysql-bin +ssl=0 + +#innodb_doublewrite=0 +#innodb_flush_log_trx_commit=0 +#innodb_checksum_algorithm=none +#innodb_log_checksums=off +#foreign_key_checks=off +#unique_checks=off +table_open_cache=15000 + +#thread_handling=loaded-dynamically + + -- 2.23.0

1 0

[PATCH compass-ci] os-cifs/*: change base image name
by Wang Yong 13 Nov '20

13 Nov '20

update cifs base image name to maintain the consistency with os-http/os-nfs Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/os-cifs/build | 2 +- container/os-cifs/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/os-cifs/build b/container/os-cifs/build index 56ebbed..77be8e9 100755 --- a/container/os-cifs/build +++ b/container/os-cifs/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t alpine/samba . +docker build -t os-cifs . diff --git a/container/os-cifs/start b/container/os-cifs/start index 1b2cd23..f42c71c 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -19,7 +19,7 @@ cmd=( -v /srv/result:/srv/result --name os-cifs --restart=always - alpine/samba + os-cifs ) "${cmd[@]}" -- 2.23.0

1 0

[PATCH v7 compass-ci 11/11] mail-robot: documentation for service
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..385ce96 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. apply-account + init my_info + - my_email + - my_name + check_to_send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0

[PATCH v7 compass-ci 10/11] mail-robot: start file
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..5827e18 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v7 compass-ci 09/11] mail-robot: build file
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v7 compass-ci 08/11] mail-robot: Dockerfile
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v7 compass-ci 07/11] mail-robot: build fail message
by Luan Shengde 13 Nov '20

13 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-fail-eamil.rb diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..769853f --- /dev/null +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + The subject should exactly: apply account + + 2. commit url + When you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + Attention: + Ensure you commit url exist and available to access. + + 3. ssh pubkey + You need to add a pubkey as an attachment to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v7 compass-ci 05/11] mail-robot: apply new account for user
by Luan Shengde 13 Nov '20

13 Nov '20

ApplyJumperAccount used to apply account for user when successfully parsed my_commit_url and my_ssh_pubkey input: my_info, my_ssh_pubkey apply_jumper_account apply account with my_info and my_ssh_pubkey my_info: - my_email - my_name - my_uuid my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/apply-jumper-account.rb | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/mail-robot/lib/apply-jumper-account.rb diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb new file mode 100755 index 0000000..a2c239c --- /dev/null +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 + +# used to apply account +# be called after AssignAccount successfully parsed my_commit_url and my_ssh_pubkey +# apply_jumper_account +# apply jumper account with my_info and my_ssh_pubkey +# account_info_exist +# check account exists +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + error_message += 'You may try again later or consulting the manager for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v7 compass-ci 04/11] mail-robot: check commit url for gitee
by Luan Shengde 13 Nov '20

13 Nov '20

GiteeCommitUrlCheck used to check commit for hub gitee.com when execute commit_url_availability in ParseApplyAccountEmail input: my_commit_url gitee_commit_check clone the repo and check commit exists and availalbe Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 container/mail-robot/lib/gitee-commit-url-check.rb diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..23b91f1 --- /dev/null +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# used to check commit available for hub gitee.com +# it is called when: +# - ParseApplyAccountEmail execute commit_url_availability +# - hub is gitee.com +# gitee_commit_check +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_check + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform whether the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with your email.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v7 compass-ci 03/11] mail-robot: parse apply account email
by Luan Shengde 13 Nov '20

13 Nov '20

ParseApplyAccountEmail used to parse my_commit_url and my_ssh_pubkey when execute check_to_send_account in ApplyAccount input: mail_content output: my_commit_url, my_ssh_pubkey parse_commit_url extract_commit_url extract my_commit_url from mail_content base_url_in_upstream_repos check whether the repo's url in upstream-repos commit_url_availability check the commit available - gitee.com call GiteeCommitUrlCheck to check commit available - non-gitee.com execute curl and check feedback for the commit parse_pub_key check attachment for my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 130 ++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100755 container/mail-robot/lib/parse-apply-account-email.rb diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..f6829c6 --- /dev/null +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -0,0 +1,130 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'gitee-commit-url-check' + +# used to parse mail_content for my_commit_url and my_ssh_pubkey +# be called by AssignAccount when it needs to extract required data: +# - my_commit_url +# - my_ssh_pubkey +# +# input: mail_content +# output: my_commit_url, my_ssh_pubkey +# +# parse_commit_url +# parse my_commit_url from the mail_content and return it +# extract my_commit_url +# extract my_coomit_url from the mail_content +# check commit url exists +# base_url_in_upstream_repos +# check whether the base url in upstream-repos +# commit_url_availability +# check commit available +# gitee.com: +# call GiteeCommitUrlCheck to check the commit +# non-gitee.com: +# execute curl to check the commit +# parse_pub_key +# check my_ssh_pubkey exists and return it +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += 'Ensure that you have add a right commit url, ' + error_message += 'and with prefix "my oss commit:".' + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit information + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_check + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not confirm the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + pub_key_exist(pub_key) + + return pub_key + end + + def pub_key_exist(pub_key) + return unless pub_key.nil? + + error_message = "No pub_key found.\n" + error_message += 'Please add a pub_key as an attachment to your email.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] os-cifs/start: change deprecated name
by Wang Yong 13 Nov '20

13 Nov '20

[problem] The original container could not be deleted due to a container name error before: docker_rm samba after: docker_rm os-cifs Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/os-cifs/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index 6cc08c6..1b2cd23 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -8,7 +8,7 @@ lsmod | grep -q "^cifs\s" || { sudo modprobe cifs } -docker_rm samba +docker_rm os-cifs cmd=( docker run -dt -- 2.23.0

1 0

[PATCH compass-ci] os-cifs/start: change deprecated name
by Wang Yong 13 Nov '20

13 Nov '20

[problem] The original container could not be deleted due to a container name [error before: docker_rm samba after: docker_rm os-cifs Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/os-cifs/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index 6cc08c6..1b2cd23 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -8,7 +8,7 @@ lsmod | grep -q "^cifs\s" || { sudo modprobe cifs } -docker_rm samba +docker_rm os-cifs cmd=( docker run -dt -- 2.23.0

1 1

[PATCH lkp-tests] hackbench-git.yaml: remove runtime
by Zhang Yale 13 Nov '20

13 Nov '20

runtime is unnecessary for hackbench-git test. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/hackbench-git.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/hackbench-git.yaml b/jobs/hackbench-git.yaml index e2320621..aad237a6 100644 --- a/jobs/hackbench-git.yaml +++ b/jobs/hackbench-git.yaml @@ -12,7 +12,6 @@ nr_threads: - 20 iterations: 10 loops: 1000 -runtime: 300 hackbench-git: mode: -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/3] iso2rootfs: solve the problem of multiple vmlinuz failures during test
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] Due to compass-ci patch: - git repo: https://gitee.com/wu_fengguang/compass-ci.git - commit id: 960aff501db5645c57459b317d0ce4dd3924f7bc As a result, when getting var(kernel_path), the vmlinuz soft link is also included, and our var(kernel_path) gets two identical vmlinuz. root cause: yuchuan@localhost rootfs_dir% ls -hl boot/vmlinuz* lrwxrwxrwx 1 root root 41 Nov 12 12:03 boot/vmlinuz -> vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 -rwxr-xr-x 1 root root 7.1M Nov 12 11:54 boot/vmlinuz-0-rescue-47178e4890764247aa6a35b8517bbc82 -rw-r--r-- 1 root root 20M Nov 12 12:04 boot/vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 yuchuan@localhost rootfs_dir% ls -hl boot/vmlinuz-* -rwxr-xr-x 1 root root 7.1M Nov 12 11:54 boot/vmlinuz-0-rescue-47178e4890764247aa6a35b8517bbc82 -rw-r--r-- 1 root root 20M Nov 12 12:04 boot/vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 [How] 1. When filtering vmlinuz: Before: realpath {rootfs_dir}/boot/vmlinuz* | grep -v rescue After: realpath {rootfs_dir}/boot/vmlinuz-* | grep -v rescue 2. Add a judgement, if the kernel num of result rootfs is not equal to 1, exit. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 05274164f8c1..c728a7e9335a 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz-*|grep -v rescue) local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + + local kernel_num=$(echo "$kernel_path" | wc -l) + [ "$kernel_num" -eq "1" ] || die "result rootfs have multiple vmlinuz, num: $kernel_num" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

2 2

[PATCH v4 lkp-tests] netperf/PKGBUILD: download new config.guess, config.sub if guess failed
by Lin Jiaxin 13 Nov '20

13 Nov '20

error: ./config.guess: unable to guess system type This script, last modified 2005-08-03, has failed to recognize the operating system you are using. It is advised that you download the most up to date version of the config scripts. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- pkg/netperf/PKGBUILD | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/pkg/netperf/PKGBUILD b/pkg/netperf/PKGBUILD index 1b724948..e4747dd5 100755 --- a/pkg/netperf/PKGBUILD +++ b/pkg/netperf/PKGBUILD @@ -10,6 +10,13 @@ md5sums=('e0d45b5bca1eee2aef0155de82366202') build() { cd "$srcdir/$pkgname-$pkgname-$pkgver.$pkgrel" + + # the orginal config.guess is too old to guess new system type + # so download latest version if it guess failed + ./config.guess > /dev/null 2>&1 || { + curl -s -o config.guess 'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.gues…' + curl -s -o config.sub 'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;…' + } # sendfile_tcp_stream() misses assignment of len and it is possible # for random len to trigger the following issue, so fix it. -- 2.23.0

1 0

[PATCH lkp-tests] stats: add libmicro result parsing file
by Bai Jing 13 Nov '20

13 Nov '20

[input] prc thr usecs/call samples errors cnt/samp getpid 1 1 0.15629 193 0 100000 [output] getpid: 0.15629 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/libmicro | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 stats/libmicro diff --git a/stats/libmicro b/stats/libmicro new file mode 100755 index 00000000..2e33ffc9 --- /dev/null +++ b/stats/libmicro @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby + +# prc thr usecs/call samples errors cnt/samp +# close_tmp 1 1 0.61880 201 0 640 + +def line_fun(line) + hash = {} + test = line.split[0] + usecs_call = line.split[3] + hash[test] = usecs_call + hash.each do |key, value| + puts "#{key}: #{value}" + end +end + +$stdin.each_line do |line| + case line + when /^\w+\s+\d/ + line_fun(line) + end +end -- 2.23.0

1 1

[PATCH v4 compass-ci] container/srv-http: support display *.log file content in browser
by Lu Weitao 13 Nov '20

13 Nov '20

[Why] Haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser, but we expect display the file content [How] 1. Add correspondence of file in mime.types 2. Set default type in default.conf of nginx, other type file that haven't define in mime.types, will display the file content in browser Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

1 0

[PATCH v7 lkp-tests] stats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..aafe6dde --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +time = [] + +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/3] iso2rootfs: allow generate a rootfs without testing it
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] If user only want generate a rootfs without testing it, he can not write submit related fields in the job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 ++ tests/iso2rootfs | 22 +++++++++++++++------- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 644c64be06b4..d4f0ffc0e80c 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,6 +11,8 @@ iso2rootfs: rootfs_path: initrd_http_host: initrd_http_port: + + # submit test yaml related fields test_yaml: test_os: test_os_arch: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index e8a027b0dad2..4f7843d7f348 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -53,15 +53,8 @@ check_yaml_vars() "rootfs_path" "initrd_http_host" "initrd_http_port" - "test_yaml" - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" ) - local yaml_t for yaml_t in "${yaml_vars[@]}" do [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." @@ -439,10 +432,25 @@ generate_submit_append_str() submit_test_job_yaml() { + [ -n ${test_yaml} ] || return 0 + log_info "starting submit ${test_yaml} ..." [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + local test_yaml_vars=( + "test_os" + "test_os_arch" + "test_os_version" + "test_os_mount" + "test_testbox" + ) + + for yaml_t in "${test_yaml_vars[@]}" + do + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + done + cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh -- 2.23.0

1 0

[PATCH v6 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..2eb0dd1 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

2 4

[PATCH v5 compass-ci] add the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 13 Nov '20

13 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result it may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container [Usage]: Two ways for the client to mount initrd: 1. initrd. mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage. mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..341bfc9 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash && \ + rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..ca9de39 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine/initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..7232bdf --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v2 lkp-tests 3/3] iso2rootfs: solve the problem of multiple vmlinuz failures during test
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] We use compass-ci/container/qcow2rootfs to generate rootfs from qcow2, but compass-ci updated the code, qcow2rootfs create a vmlinuz soft link at {rootfs_dir}/boot. As a result, when getting var(kernel_path), the vmlinuz soft link is also included, and our var(kernel_path) gets two identical vmlinuz. Error msg: + sed -i 's|KERNEL|"{rootfs_dir}/boot/vmlinuz-4.19.90-2010.2.0.0046.oe1.aarch64 {rootfs_dir}/boot/vmlinuz-4.19.90-2010.2.0.0046.oe1.aarch64"|g' {i2r_src}/env/iso2rootfs/kvm.sh sed: -e expression #1, char 133: unterminated `s' command [How] 1. When filtering vmlinuz: Before: realpath {rootfs_dir}/boot/vmlinuz* | grep -v rescue After: realpath {rootfs_dir}/boot/vmlinuz-* | grep -v rescue 2. Add a judgement, if the kernel num of result rootfs is not equal to 1, exit. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 87a51acaca5f..754638c95b39 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz-*|grep -v rescue) local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + + local kernel_num=$(echo "$kernel_path" | wc -l) + [ "$kernel_num" == "1" ] || die "result rootfs have multiple vmlinuz, num: $kernel_num" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

3 4

[PATCH v2 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 13 Nov '20

13 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971s response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..8eaac442 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,28 @@ +#!/usr/bin/env ruby + +LKP_SRC ||= ENV['LKP_SRC'] + +# General statistics: +# total time: 0.3987s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +def add_result(results, key, val) + results[key] ||= [] + results[key] << val +end + +$stdin.each_line do |line| + case line + when /total time:\s+([0-9.]+)/ + add_result(results_total, 'total time', $1.to_f) + end +end + +results_total.each do |key, vals| + puts "#{key}: #{vals.inject(0.0, :+)s}" +end -- 2.23.0

3 2

[PATCH v3 compass-ci] container/srv-http: support display *.log file content in browser
by Lu Weitao 13 Nov '20

13 Nov '20

[why] haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser now. [how] 1. add correspondence of file in mime.types 2. set default type in default.conf of nginx, other type file that haven't define in mime.types, will display the file content in browser Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

4 7

[PATCH lkp-tests] lib: add multi-args in job yaml
by Bai Jing 13 Nov '20

13 Nov '20

[original style] os: openeuler os_verison: 20.03 os_arch: aarch64 os: centos os_version: 8.7 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - centos 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file @@ -703,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

4 7

[PATCH v4 lkp-tests] stats: add sysbench-mysql result parsing file
by Zhang Yu 13 Nov '20

13 Nov '20

[input] SQL statistics: queries performed: total: 298163456 transactions: 14908172 (25024.26 per sec.) [output] transactions: 25024.26 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/sysbench-mysql | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 stats/sysbench-mysql diff --git a/stats/sysbench-mysql b/stats/sysbench-mysql new file mode 100755 index 00000000..836ab6c9 --- /dev/null +++ b/stats/sysbench-mysql @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + case line.chomp! + when /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix LAB -> lab
by Ren Wen 13 Nov '20

13 Nov '20

1) Global config uses "lab" not "LAB". 2) Do a crystal-format. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..c263478 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -10,7 +10,7 @@ JOB_ES_PORT = 9200 JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" -LAB = (ENV.has_key?("LAB") ? ENV["LAB"] : "nolab") +LAB = (ENV.has_key?("lab") ? ENV["lab"] : "nolab") SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 @@ -25,7 +25,7 @@ SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" -OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" -SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" +OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" +SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" DEMO_JOB = %({"suite":"pixz","testcase":"pixz","category":"benchmark","nr_threads":1,"pixz":null,"job_origin":"jobs/pixz.yaml","testbox":"wfg-e595","arch":"x86_64","tbox_group":"wfg-e595","id":"100","kmsg":null,"boot-time":null,"uptime":null,"iostat":null,"heartbeat":null,"vmstat":null,"numa-numastat":null,"numa-vmstat":null,"numa-meminfo":null,"proc-vmstat":null,"proc-stat":null,"meminfo":null,"slabinfo":null,"interrupts":null,"kconfig":"x86_64-rhel-7.6","compiler":"gcc-7"}) -- 2.23.0

1 0

[PATCH v7 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 13 Nov '20

13 Nov '20

[Why] hackbench-git is an old version tool required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..f2be0c9d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,30 @@ +#!/bin/sh +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is both a benchmark and a stress test for the Linux kernel scheduler. +## Its main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes)which communicate via either sockets +## or pipes and time how long it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], loops, ipc [pipe/socket] + hackbench_path="$BENCHMARK_ROOT"/hackbench-git/hackbench/hackbench + [ ! -x "${hackbench_path}" ] && exit 1 + + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

1 1

[PATCH v3 lkp-tests] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 13 Nov '20

13 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..093bbe4d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,19 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -86,24 +99,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -} -create_softlink() -{ - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" + [ "$?" == 0 ] || create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats: add hackbench-git result parsing script.
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..187f1c19 --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats: add sysbench-mysql result parsing file
by Zhang Yu 13 Nov '20

13 Nov '20

[input] SQL statistics: queries performed: total: 298163456 transactions: 14908172 (25024.26 per sec.) ... [output] transactions: 25024.26 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/sysbench-mysql | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100755 stats/sysbench-mysql diff --git a/stats/sysbench-mysql b/stats/sysbench-mysql new file mode 100755 index 00000000..5f4e9fd6 --- /dev/null +++ b/stats/sysbench-mysql @@ -0,0 +1,34 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) +# +# General statistics: +# total time: 600.0222s +# total number of events: 15015147 +# +# Latency (ms): +# min: 2.74 +# avg: 10.23 +# max: 2238.49 +# 95th percentile: 15.55 +# sum: 153573603.78 +# +# Threads fairness: +# events (avg/stddev): 58652.9180/4292.14 +# execution time (avg/stddev): 599.8969/0.01 + +while (line = STDIN.gets) + case line.chomp! + when /transactions:[^(]*\(([^\s]*)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

2 2

[PATCH lkp-tests] distro/archLinux: add archLinux distro file
by Wang Yong 13 Nov '20

13 Nov '20

add archLinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archLinux | 103 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100755 distro/archLinux diff --git a/distro/archLinux b/distro/archLinux new file mode 100755 index 000000000..152f24ff3 --- /dev/null +++ b/distro/archLinux @@ -0,0 +1,103 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs $pkg || pacman -Sy $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find /var/ -type f -name "*.pkg.tar.xz") + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar.gz > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +cleanup_downloaded_pkgs() +{ + find /var/ -type f -name "*.pkg.tar.xz" -exec rm -f -- '{}' \; +} + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + fi + done +} -- 2.23.0

3 4

[PATCH v6 lkp-tests] stats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..4ed6a0bc --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,26 @@ +#!/usr/bin/env ruby + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 0