- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to separate job.yaml, it contains several yaml documents separated by "---" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 19 +++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 27 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 104 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..fbb9dc55 --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,19 @@ +suite: ftq + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..f3e2c553 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: +monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..63bac5cf --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: +monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..a802dc1b --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index a83d2102..169d7812 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -44,4 +44,8 @@ describe 'submit job spec' do it 'link matrix' do submit_job('matrix') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

1 0

[PATCH v4 lkp-tests] setup/sysctl: add system parameters
by Zhang Yu 26 Nov '20

26 Nov '20

[why] When testing need to setup system parameters. [how] Add parameters to setup/sysctl. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- setup/sysctl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/sysctl b/setup/sysctl index 81374939..2ace4db2 100755 --- a/setup/sysctl +++ b/setup/sysctl @@ -202,6 +202,9 @@ # - sc_netdev_max_backlog # - sc_rmem_max # - sc_wmem_max +# - sc_ipv4_tcp_rmem +# - sc_ipv4_tcp_wmem +# - sc_sched_features . $LKP_SRC/lib/reproduce-log.sh @@ -405,8 +408,11 @@ [ -n "$sc_ipv4_tcp_max_tw_buckets" ] && log_eval "echo '$sc_ipv4_tcp_max_tw_buckets' > '/proc/sys/net/ipv4/tcp_max_tw_buckets'" [ -n "$sc_ipv4_tcp_max_syn_backlog" ] && log_eval "echo '$sc_ipv4_tcp_max_syn_backlog' > '/proc/sys/net/ipv4/tcp_max_syn_backlog'" [ -n "$sc_ipv4_ip_local_port_range" ] && log_eval "echo '$sc_ipv4_ip_local_port_range' > '/proc/sys/net/ipv4/ip_local_port_range'" +[ -n "$sc_ipv4_tcp_rmem" ] && log_eval "echo '$sc_ipv4_tcp_rmem' > '/proc/sys/net/ipv4/tcp_rmem'" +[ -n "$sc_ipv4_tcp_wmem" ] && log_eval "echo '$sc_ipv4_tcp_wmem' > '/proc/sys/net/ipv4/tcp_wmem'" [ -n "$sc_netdev_max_backlog" ] && log_eval "echo '$sc_netdev_max_backlog' > '/proc/sys/net/core/netdev_max_backlog'" [ -n "$sc_rmem_max" ] && log_eval "echo '$sc_rmem_max' > '/proc/sys/net/core/rmem_max'" [ -n "$sc_wmem_max" ] && log_eval "echo '$sc_wmem_max' > '/proc/sys/net/core/wmem_max'" +[ -n "$sc_sched_features" ] && log_eval "echo '$sc_sched_features' > '/sys/kernel/debug/sched_features'" exit 0 -- 2.23.0

1 0

[PATCH v3 compass-ci] web_backend.rb: fix : compare_groups_matrices needs arg suites_list
by Li Yuanchao 26 Nov '20

26 Nov '20

compare_groups_matrices needs three args, but now only get two, lost suites_list. divide method to pass rubocop limit Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..90c8a58 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -147,6 +147,20 @@ def get_es_must(params) must end +def _get_group_matrices(query, dimension) + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + while matrices.empty? + from += size + break if from > total + + query[:from] = from + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + end + [matrices, suites_list] +end + def get_groups_matrices(conditions, dimension, must, size, from) must += build_mutli_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } @@ -166,27 +180,17 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - while matrices.empty? - from += size - break if from > total - - query[:from] = from - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - end - matrices + matrices, suites_list = _get_groups_matrices(query, dimension) end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: about multi-device test
by Ren Wen 26 Nov '20

26 Nov '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- doc/multi-device-test.md | 109 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 doc/multi-device-test.md diff --git a/doc/multi-device-test.md b/doc/multi-device-test.md new file mode 100644 index 0000000..1be8c79 --- /dev/null +++ b/doc/multi-device-test.md @@ -0,0 +1,109 @@ +# Multi-device Test + +在进行需要多台机器的测试的时候, 需要搭建多台机器的环境. + +## When using multi-device + + * 当被测试程序需要启动一个守护进程时 + * 当执行测试时需要访问另外的机器时 + +## How to run a multi-device test + +在要提交的job.yaml中, 需要一个**cluster**字段, 它的值是一个多机环境的配 +置文件的文件名, 配置文件的内容可以参考: *lab-z9/cluster/*. +调度器会根据配置文件的内容将该job拆分为对应机器数量的job, 然后分别提交到 +相应的队列中. + +那么如何对指定的机器执行指定的测试呢? +在job.yaml中增加这样的内容: + +``` yaml +if role <role01> + <test_script01>: + <test_script02>: + +if role <role02> + <test_script03>: + <test_script04>: + +... +``` + +在多机的配置文件中, 会给每台机器赋予一个角色. 当每台机器取得自己队列中的 +job时, 按照自己的角色去执行相应角色所对应的脚本. + +比如说: 一台多机测试中的机器的角色为*role01*, 那么它会去寻找脚本 +*test_script01*, *test_script02*, 然后去执行. + +### how to find test script + +在lkp-tests的**daemon**, **tests**目录下查找这些脚本. +假设在**daemon**下找到了*test_script01*, 则将该脚本作为`start_daemon`的参 +数运行. 如果在**tests**下找到了*test_script01*, 则作为`run_test`的参数去 +运行. 如果没有找到相应的文件, 则按变量导出. + +### about `start_daemon` and `run_test` + +用于启动脚本的这2个函数位于**lkp-tests/lib/job.sh**. +要搞清楚多机的流程, 就得明白这两个函数是怎么去做的. + +#### the processes of `start_damon` + + 1. 运行在**daemon**文件夹下找到的脚本. 如果运行成功, 向调度器报告一个状态: + `started`, 调度器记录该机器的状态为: `start`; 执行失败, 则报告: + `failed`, 调度器记录为: `abort`, 然后退出执行. 状态可以查看redis的 + key: `HGETALL sched/cluster_state`. + 2. 向调度器报告一个状态: `write_state`, 并携带一些机器相关的参数: + * `node_roles`: 机器的角色 + * `ip`: 通过命令`hostname -I`取得的第一个值 + * `direct_macs`: 多机配置文件中该机器的**macs**字段 + * `direct_ips`: 调度器根据多机配置文件中的**ip0**字段配置的直连ip, 与 + 配置文件中**macs**的值一一对应 + 此时调度器将这些参数记录为每台机器相关的信息, 可以查看redis的key: + **sched/cluster_state**. + 3. 将**direct_ips**的值一一绑定到机器中的**direct_macs**的相应设备上. + 4. 向调度器报告一个状态: `wait_ready`, 调度器将其记录为: `ready`; 与此 + 同时, 调度器查看是否多机测试中的所有机器都准备好了. 如果每一台机器 + 都准备好了, 则返回`ready`; 如果其中一台的状态为`abort`, 则返回 + `abort`, 然后退出执行; 其余情况, 则循环查询, 直到前2种情形发生. + 5. 向调度器报告一个状态: `roles_ip`, 并将调度器的返回值导出为环境变量. + 比如当返回值为: `ip=ip01`时, 执行: `export ip=ip01`. + 6. 向调度器报告一个状态: `wait_finish`, 调度器将其记录为: `finish`; 与 + 此同时, 调度器查看是否多机测试中的所有机器的状态都为`finish`. 如果 + 都是`finish`, 则返回`finish`; 如果其中一台的状态为`abort`, 则返回 + `abort`, 然后退出执行; 其余情况, 则循环查询, 直到前2种情形发生. + 7. 执行完成. + +#### the processes of `run_test` + +步骤1,2,3,4同上述2,3,4,5. + + 1. 向调度器报告一个状态: `write_state`, 并携带一些机器相关的参数: + * `node_roles`: 机器的角色 + * `ip`: 通过命令`hostname -I`取得的第一个值 + * `direct_macs`: 多机配置文件中该机器的**macs**字段 + * `direct_ips`: 调度器根据多机配置文件中的**ip0**字段配置的直连ip, 与 + 配置文件中**macs**的值一一对应 + 此时调度器将这些参数记录为每台机器相关的信息, 可以查看redis的key: + **sched/cluster_state**. + 2. 将**direct_ips**的值一一绑定到机器中的**direct_macs**的相应设备上. + 3. 向调度器报告一个状态: `wait_ready`, 调度器将其记录为: `ready`; 与此 + 同时, 调度器查看是否多机测试中的所有机器都准备好了. 如果每一台机器 + 都准备好了, 则返回`ready`; 如果其中一台的状态为`abort`, 则返回 + `abort`, 然后退出执行; 其余情况, 则循环查询, 直到前2种情形发生. + 4. 向调度器报告一个状态: `roles_ip`, 并将调度器的返回值导出为环境变量. + 比如当返回值为: `ip=ip01`时, 执行: `export ip=ip01` + 5. 运行在**tests**文件夹下找到的脚本. 运行完成, 向调度器报告一个状态: + `finished`, 调度器记录为: `finish`; 执行失败, 则报告: `failed`, 调 + 度器记录为: `abort`, 然后退出执行. + 6. 执行完成. + +#### others + +`start_daemon`和`run_test`还会报告一个`abort`状态: 循环100次向调度器发 +起请求, 查看所有机器是否全部`ready`或`finish`, 如果循环结束仍然返回的是 +`retry`, 则报告该状态, 并结束执行. + +当调度器接受到与之相关的请求时, 会阻塞(或循环)查询这些机器的状态, 因此只 +需返回`ready`(`finish`)或`abort`. 所以不会触发循环100次请求这个过程, 因 +此不会报告`abort`这个状态. -- 2.23.0

1 0

[PATCH lkp-tests] distro/*: add exit when failed download packages
by Wang Yong 26 Nov '20

26 Nov '20

now when download packages failed, echo error message and $? is 0, so add exit 1 to get last_state Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 1 + distro/archlinux | 1 + distro/debian | 1 + 3 files changed, 3 insertions(+) diff --git a/distro/aliyun b/distro/aliyun index 79656f0e4..598243d8e 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -126,6 +126,7 @@ pack_benchmark_deps() yum_pack else echo "failed to pack-deps $BM_NAME" >&2 + exit 1 fi done } diff --git a/distro/archlinux b/distro/archlinux index 3f8d62ebc..3942080a6 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -24,6 +24,7 @@ pack_benchmark_deps() pacman_pack else echo "failed to pack-deps $BM_NAME" >&2 + exit 1 fi done } diff --git a/distro/debian b/distro/debian index 13c4e983a..8821e3ddd 100755 --- a/distro/debian +++ b/distro/debian @@ -271,6 +271,7 @@ pack_benchmark_deps() echo "$PACKAGE_LIST" > $pack_to/.${BM_NAME}.packages else echo "failed to pack-deps $BM_NAME" >&2 + exit 1 fi rm -fr "$work_dir" -- 2.23.0

1 0

[PATCH compass-ci 1/4] answerback-email: add new options
by Luan Shengde 26 Nov '20

26 Nov '20

add option for answerback-email tool -n|--name name specify my_name for user when assigning account -l|--login Y|y|N|n enable/disable user login permission -u|--update enable update configuration for assigned account -u option enable update for: my_name my_ssh_pubkey generate sshkey on jumper reset user's login permission Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 147 +++++++++++++++---- 1 file changed, 120 insertions(+), 27 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 5ddc451..bffba17 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -14,6 +14,7 @@ require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' require_relative 'build-send-account-email' +require_relative 'build-update-email' names = Set.new %w[ JUMPER_HOST @@ -33,25 +34,61 @@ my_info = { 'my_email' => nil, 'my_name' => nil, 'my_commit_url' => nil, - 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => [], - 'gen_sshkey' => false + 'my_uuid' => nil, + 'my_login_name' => nil, + 'my_ssh_pubkey' => [] } -def init_info(mail_content, my_info) +# stdin_info is used to store infos added with option: +# -e email_address +# -n name +# -s pubkey_file +# email_info is used to store infos parsed from email_file with option: +# -f email_file +# my_info_es is used to store info read from ES when update user +# when use --update option +# stdin_info, email_info, my_info_es have different priority +# stdin_info > email_info > my_info_es +# when assigning account or update conf for account +# if they have the same key, my_info will use the value with higher priority. +# conf_info is used to store keys used to config the account +stdin_info = {} +email_info = {} +my_info_es = {} +conf_info = { + 'gen_sshkey' => false, + 'enable_login' => true, + 'is_update_account' => false +} + +def init_info(mail_content, email_info, my_info) my_info['my_email'] = mail_content.from[0] - my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + email_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') return if mail_content.attachments.empty? - my_info['my_ssh_pubkey'] << mail_content.attachments[0].body.decoded + email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip +end + +def read_my_login_name(my_email, my_info_es) + my_account_info_str = %x(curl -XGET localhost:9200/accounts/_doc/#{my_email}) + my_account_info = YAML.safe_load my_account_info_str + message = "No such email found from the ES: #{my_email}" + raise message unless my_account_info['found'] + + my_info_es.update my_account_info['_source'] end options = OptionParser.new do |opts| - opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey]\n" - opts.banner += " -e or -f is required\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|Y|n|N] [--update]\n" + opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' + opts.banner += "[-g|--gen-sshkey] [--login y|Y|n|N] [--update]\n" + opts.banner += " -e|-f is required when applying account or updating account\n" + opts.banner += " -n is require when assigning account with -e\n" opts.banner += " -s is optional when use -e\n" - opts.banner += ' -g is optional, used to generate sshkey for user' + opts.banner += " -g is optional, used to generate sshkey for user\n" + opts.banner += " -u is require when updating an account\n" + opts.banner += ' -l is optional, used to enable/disable login permission' opts.separator '' opts.separator 'options:' @@ -60,18 +97,37 @@ options = OptionParser.new do |opts| my_info['my_email'] = email_address end + opts.on('-n name', '--name name', 'appoint name') do |name| + stdin_info['my_name'] = name + end + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp + stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| mail_content = Mail.read(email_file) - init_info(mail_content, my_info) + init_info(mail_content, email_info, my_info) end opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do - my_info['gen_sshkey'] = true + conf_info['gen_sshkey'] = true + end + + opts.on('-u', '--update', 'updata user account') do + read_my_login_name(my_info['my_email'], my_info_es) + conf_info['is_update_account'] = true + end + + opts.on('-l Y|y|N|n', '--login Y|y|N|n', 'enable/disable user login permission') do |value| + if %w[y Y].include? value + conf_info['enable_login'] = true + elsif %w[n N].include? value + conf_info['enable_login'] = false + else + raise 'invalid parameter, please use y|Y|n|N' + end end opts.on_tail('-h', '--help', 'show this message') do @@ -82,33 +138,70 @@ end options.parse!(ARGV) -def apply_account(my_info) - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') +def apply_account(my_info, conf_info) + apply_info = {} + apply_info.update my_info + apply_info.update conf_info + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account(my_info) +def check_my_email(my_info) message = "No email address specified\n" - message += "use -e to add a email address\n" + message += "use -e to add a email address for applying account\n" message += 'or use -f to add a email file' - raise message if my_info['my_email'].nil? - account_info = apply_account(my_info) - my_info['my_login_name'] = account_info['my_login_name'] + raise message if my_info['my_email'].empty? +end +def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) + new_email_pubkey = email_info.delete 'new_email_pubkey' + new_stdin_pubkey = stdin_info.delete 'new_ssh_pubkey' + new_pubkey = new_stdin_pubkey || new_email_pubkey + + my_info.update my_info_es unless my_info_es.empty? + my_info.update email_info unless email_info.empty? + my_info.update stdin_info unless stdin_info.empty? + + return if new_pubkey.nil? + return if my_info['my_ssh_pubkey'].include? new_pubkey + + my_info['my_ssh_pubkey'].insert(0, new_pubkey) +end + +def build_my_info_from_account_info(my_info, account_info, conf_info) unless account_info['my_jumper_pubkey'].nil? - my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + return if my_info['my_ssh_pubkey'][-1] == account_info['my_jumper_pubkey'] + + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] end - my_info.delete 'gen_sshkey' + my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] +end + +def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) + check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] + build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) + + message = 'No my_name found, please use -n to add one' + raise message if my_info['my_name'].nil? + + account_info = apply_account(my_info, conf_info) + build_my_info_from_account_info(my_info, account_info, conf_info) + store_account_info(my_info) - send_mail(my_info, account_info) + send_mail(my_info, account_info, conf_info) end -def send_mail(my_info, account_info) - message = build_message(my_info['my_email'], account_info) - +def send_mail(my_info, account_info, conf_info) + message = if conf_info['is_update_account'] + build_update_message(my_info['my_email'], account_info, conf_info) + else + build_message(my_info['my_email'], account_info) + end %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end @@ -117,4 +210,4 @@ def store_account_info(my_info) es.put_source_by_id(my_info['my_email'], my_info) end -send_account(my_info) +send_account(my_info, conf_info, email_info, my_info_es, stdin_info) -- 2.23.0

2 2

[PATCH compass-ci] web_backend.rb: fix : compare_groups_matrices needs arg suites_list
by Li Yuanchao 26 Nov '20

26 Nov '20

compare_groups_matrices needs three args, but now only get two, lost suites_list. divide method to pass rubocop limit Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..b5b06b4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -147,6 +147,20 @@ def get_es_must(params) must end +def _get_group_matrices(query, dimension) + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + while matrices.empty? + from += size + break if from > total + + query[:from] = from + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + end + [matrices, suites_list] +end + def get_groups_matrices(conditions, dimension, must, size, from) must += build_mutli_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } @@ -166,27 +180,18 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - while matrices.empty? - from += size - break if from > total - - query[:from] = from - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - end - matrices + matrices, suites_list = _get_groups_matrices(query, dimension) + [matrices, suites_list] end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH v3 lkp-tests] submit: fix the processes of option: --my-queue
by Ren Wen 26 Nov '20

26 Nov '20

If use options '--my-queue' and '-o' at the same time, the output job file will not contain the value of the covered field 'queue'. So pass the 'jobs' not 'job_hash'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index be588bed..23518eb0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -112,6 +112,7 @@ ARGV.each do |jobfile| jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') + add_my_queue(jobs) if opt_my_queue jobs['node_roles'] ||= 'server client' if jobs['cluster'] jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] @@ -144,7 +145,6 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) - add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/job: add category.md
by Zhang Yu 26 Nov '20

26 Nov '20

Add a document for introducing the category field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/category.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/job/category.md diff --git a/doc/job/category.md b/doc/job/category.md new file mode 100644 index 0000000..6425318 --- /dev/null +++ b/doc/job/category.md @@ -0,0 +1,21 @@ +# category + +Meaning: +- Each test has its category, the category field is a test type of test cases. +- When you write a job yaml, you can choose one of benchmark, functional, noise-benchmark + to assign the value of category. +- Here are some differences about different category values: + 1. If category is benchmark, it will monitor some system information: + kmsg: print the kernel startup information + boot-time: start time diagnosis item + uptime: display how long the system has been running + iostat: print the time when the data is collected + heartbeat: provide the heartbeat monitors service + ... + other monitors you can see them in $LKP_SRC/include/category/benchmark + 2. If category is functional, it will monitor some system information: + kmsg: print the kernel startup information + heartbeat: provide the heartbeat monitors service + meminfo: solid memory usage status or vm usage status + ... + you can see the monitors in $LKP_SRC/include/category/functional -- 2.23.0

1 0

[PATCH v2 lkp-tests] submit: fix the processes of option: --my-queue
by Ren Wen 26 Nov '20

26 Nov '20

If use options '--my-queue' and '-o' at the same time, the output job file will not contain the value of the covered field 'queue'. So pass the 'jobs' not 'job_hash'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index be588bed..9510d53d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -108,6 +108,7 @@ ARGV.each do |jobfile| jobs = Job2sh.new jobs.cmdline_defaults = opt_cmdline_defaults jobs.overrides = opt_set_key_value + add_my_queue(jobs) if opt_my_queue jobs.load(jobfile, true) || next jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] @@ -144,7 +145,6 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) - add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

2 2

[PATCH compass-ci] doc/job/: add lkp_initrd_user.md
by Zhang Yale 26 Nov '20

26 Nov '20

Add a document for introducing the lkp_initrd_user field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/lkp_initrd_user.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/job/lkp_initrd_user.md diff --git a/doc/job/lkp_initrd_user.md b/doc/job/lkp_initrd_user.md new file mode 100644 index 0000000..b4a0982 --- /dev/null +++ b/doc/job/lkp_initrd_user.md @@ -0,0 +1,10 @@ +# lkp_inird_user + +Meaning: +- "lkp_initrd_user" is one of the init_field, the default value is "latest". +- User can set lkp_initrd_user to $USER or the default value in "/home/$USER/.config/compass-ci/default/$USER.yaml. +- If the user's lkp_tests code is changed, build & run the lkp_initrd container which in "${CCI_SRC}/container/lkp-initrd" + to pack a new lkp-${os_arch}.cgz. +- The new lkp-${os_arch}.cgz file is generated in the "${SRV_INITRD}/lkp/${lkp_initrd_user}/lkp-${os_arch}.cgz". +- When user submit one job, qemu-pxe or other testbox start mount the <"lkp"> initrd to consume the user submited <"job"> + "initrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/lkp/${lkp_initrd_user}/lkp-${os_arch}.cgz". -- 2.23.0

3 5

[PATCH compass-ci 1/5] openeuler: configure files about build image with one comman
by Wang Chenglong 26 Nov '20

26 Nov '20

[Contains] openeule-${os_version}.repo: yum configuration files files-to-exclude: list of files that need to be excluded packages-to-install: list of packages that need to install README.md: descriptive documentation Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/README.md | 15 +++++++ container/osimage/openeuler/files-to-exclude | 11 +++++ .../openeuler/openeuler-20.03-lts.repo | 43 +++++++++++++++++++ .../osimage/openeuler/openeuler-20.09.repo | 43 +++++++++++++++++++ .../osimage/openeuler/packages-to-install | 10 +++++ 5 files changed, 122 insertions(+) create mode 100644 container/osimage/openeuler/README.md create mode 100644 container/osimage/openeuler/files-to-exclude create mode 100644 container/osimage/openeuler/openeuler-20.03-lts.repo create mode 100644 container/osimage/openeuler/openeuler-20.09.repo create mode 100644 container/osimage/openeuler/packages-to-install diff --git a/container/osimage/openeuler/README.md b/container/osimage/openeuler/README.md new file mode 100644 index 0000000..cea68f9 --- /dev/null +++ b/container/osimage/openeuler/README.md @@ -0,0 +1,15 @@ +# Use the tool to create a new openeuler-${os_version} initramfs image. + +Usage: + cd ${CCI_SRC}/rootfs/initramfs/openeuler/aarch64/${os_version}/ + ./build + +Some configuration items: +./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + +./files-to-exclude + If you want remove some unnecessary files, you can write the names in ./files-to-exclude + +$HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. diff --git a/container/osimage/openeuler/files-to-exclude b/container/osimage/openeuler/files-to-exclude new file mode 100644 index 0000000..3854947 --- /dev/null +++ b/container/osimage/openeuler/files-to-exclude @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/files-to-exclude +/tmp/packages-to-install diff --git a/container/osimage/openeuler/openeuler-20.03-lts.repo b/container/osimage/openeuler/openeuler-20.03-lts.repo new file mode 100644 index 0000000..691f438 --- /dev/null +++ b/container/osimage/openeuler/openeuler-20.03-lts.repo @@ -0,0 +1,43 @@ +#generic-repos is licensed under the Mulan PSL v2. +#You can use this software according to the terms and conditions of the Mulan PSL v2. +#You may obtain a copy of Mulan PSL v2 at: +# http://license.coscl.org.cn/MulanPSL2 +#THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR +#IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR +#PURPOSE. +#See the Mulan PSL v2 for more details. + +[OS] +name=OS +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/OS/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/OS/$basearch/RPM-GPG-KEY-openEuler + +[everything] +name=everything +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/everything/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/everything/$basearch/RPM-GPG-KEY-openEuler + +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/EPOL/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/EPOL/$basearch/RPM-GPG-KEY-openEuler + +[debuginfo] +name=debuginfo +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/debuginfo/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/debuginfo/$basearch/RPM-GPG-KEY-openEuler + +[source] +name=source +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/source/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/source/RPM-GPG-KEY-openEuler diff --git a/container/osimage/openeuler/openeuler-20.09.repo b/container/osimage/openeuler/openeuler-20.09.repo new file mode 100644 index 0000000..1da7571 --- /dev/null +++ b/container/osimage/openeuler/openeuler-20.09.repo @@ -0,0 +1,43 @@ +#generic-repos is licensed under the Mulan PSL v2. +#You can use this software according to the terms and conditions of the Mulan PSL v2. +#You may obtain a copy of Mulan PSL v2 at: +# http://license.coscl.org.cn/MulanPSL2 +#THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR +#IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR +#PURPOSE. +#See the Mulan PSL v2 for more details. + +[OS] +name=OS +baseurl=http://repo.openeuler.org/openEuler-20.09/OS/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/OS/$basearch/RPM-GPG-KEY-openEuler + +[everything] +name=everything +baseurl=http://repo.openeuler.org/openEuler-20.09/everything/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/everything/$basearch/RPM-GPG-KEY-openEuler + +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/openEuler-20.09/EPOL/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/EPOL/$basearch/RPM-GPG-KEY-openEuler + +[debuginfo] +name=debuginfo +baseurl=http://repo.openeuler.org/openEuler-20.09/debuginfo/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/debuginfo/$basearch/RPM-GPG-KEY-openEuler + +[source] +name=source +baseurl=http://repo.openeuler.org/openEuler-20.09/source/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/source/RPM-GPG-KEY-openEuler diff --git a/container/osimage/openeuler/packages-to-install b/container/osimage/openeuler/packages-to-install new file mode 100644 index 0000000..f17eee1 --- /dev/null +++ b/container/osimage/openeuler/packages-to-install @@ -0,0 +1,10 @@ +cpio +systemd-udev.aarch64 +cifs-utils +nfs-utils +hostname +curl +openssh-server +openssl +iputils +wget -- 2.23.0

3 4

[PATCH v1 lkp-tests] spec/submit: add job yaml for multi-args test
by Bai Jing 26 Nov '20

26 Nov '20

[test cmd] respec $LKP_SRC/spec/submit_spec.rb [output] lkp-tests/spec/submit/matrices-args/unixbench-spec.yaml => /lkp-tests/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml lkp-tests/spec/submit/matrices-args/unixbench-spec.yaml => /lkp-tests/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrices-args/unixbench-spec.yaml | 8 ++++++ ...unixbench-spec-1-300-whetstone-double.yaml | 25 +++++++++++++++++++ .../unixbench-spec-3-300-shell1.yaml | 25 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 4 files changed, 62 insertions(+) create mode 100644 spec/submit/matrices-args/unixbench-spec.yaml create mode 100644 spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml diff --git a/spec/submit/matrices-args/unixbench-spec.yaml b/spec/submit/matrices-args/unixbench-spec.yaml new file mode 100644 index 00000000..a763927b --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec.yaml @@ -0,0 +1,8 @@ +suite: unixbench + +runtime: 300s + +nr_task test: + - 1 whetstone-double + - 3 shell1 +unixbench: diff --git a/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml new file mode 100644 index 00000000..16af43d2 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: +monitors: {} diff --git a/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml new file mode 100644 index 00000000..5c0526a1 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: +monitors: {} diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..3a9b0865 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'link matrices-args' do + submit_job('matrices-args') + end end -- 2.23.0

2 2

[PATCH lkp-tests] tats/crystal: add hackbench-git.cr
by Zhang Yale 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/crystal/hackbench-git.cr | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 stats/crystal/hackbench-git.cr diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr new file mode 100644 index 00000000..214dcd02 --- /dev/null +++ b/stats/crystal/hackbench-git.cr @@ -0,0 +1,23 @@ +#!/usr/bin/env crystal + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +time = [] of Float64 + +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(" ", "_") + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

2 2

[PATCH compass-ci 4/5] openeuler: build openeuler initramfs image with one command
by Wang Chenglong 26 Nov '20

26 Nov '20

[Why] Use the tool to create a new openeuler-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./packages-to-install. [How] We create a link button for the user to use. $CCI_SRC/rootfs/initramfs/openeuler/aarch64/${os_version}/build Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/run | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100755 container/osimage/openeuler/run diff --git a/container/osimage/openeuler/run b/container/osimage/openeuler/run new file mode 100755 index 0000000..c92c59f --- /dev/null +++ b/container/osimage/openeuler/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +export_root_passwd +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/files-to-exclude:/tmp/files-to-exclude + -v $DIR/${image}.repo:/etc/yum.repos.d/openEuler.repo + -e ROOT_PASSWD=$ROOT_PASSWD + -e IMAGE_NAME=$IMAGE_NAME + $image + /root/bin/create-image +) + +pull_docker_image +"${cmd[@]}" +docker_cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

3 6

[PATCH compass-ci 2/2] mail-robot: add sched info
by Luan Shengde 26 Nov '20

26 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/mail-robot/start b/container/mail-robot/start index 5827e18..1c55dd3 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -12,6 +12,8 @@ names = Set.new %w[ JUMPER_HOST JUMPER_PORT SEND_MAIL_PORT + SCHED_HTTPS_HOST + SCHED_HTTPS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: add sched info in email message
by Luan Shengde 26 Nov '20

26 Nov '20

for user to add the sched info in the default yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b793232..4754c4c 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -25,6 +25,8 @@ def build_apply_account_email(my_info) my_email: #{my_info['my_email']} my_name: #{my_info['my_name']} my_uuid: #{my_info['my_uuid']} + SCHED_HTTPS_HOST: #{ENV['SCHED_HTTPS_HOST']} + SCHED_HTTPS_PORT: #{ENV['SCHED_HTTPS_PORT']} EOF 2) download lkp-tests and dependencies -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job/: add category.md
by Zhang Yu 26 Nov '20

26 Nov '20

Add a document for introducing the category field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/category.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/job/category.md diff --git a/doc/job/category.md b/doc/job/category.md new file mode 100644 index 0000000..bfe80fe --- /dev/null +++ b/doc/job/category.md @@ -0,0 +1,21 @@ +# category + +Meaning: +- Every test has its category, the category field is a test type of test cases. +- When you write a job yaml, you can chose one of benchmark, functional, noise-benchmark + to assign the value of category. +- Here are the difference of difference category: + 1. If category is benchmark, it will monitor some system information: + kmsg: print the kernel startup information + boot-time: start time diagnosis item + uptime: displays how long the system has been running + iostat: print the time when the data is collected + heartbeat: provide the heartbeat monitors service + ... + other monitors you can see them in $LKP_SRC/include/category/benchmark + 2. If category is functional, it will monitor some system information: + kmsg: print the kernel startup information + heartbeat: provide the heartbeat monitors service + meminfo: solid memory usage status or vm usage status + ... + you can see the monitors in $LKP_SRC/include/category/functional -- 2.23.0

1 0

[PATCH v3 lkp-tests] etup/sysctl: add system parameters
by Zhang Yu 26 Nov '20

26 Nov '20

[why] When testing need to setup system parameters. [how] Add parameters to setup/sysctl. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- setup/sysctl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/sysctl b/setup/sysctl index 81374939..2ace4db2 100755 --- a/setup/sysctl +++ b/setup/sysctl @@ -202,6 +202,9 @@ # - sc_netdev_max_backlog # - sc_rmem_max # - sc_wmem_max +# - sc_ipv4_tcp_rmem +# - sc_ipv4_tcp_wmem +# - sc_sched_features . $LKP_SRC/lib/reproduce-log.sh @@ -405,8 +408,11 @@ [ -n "$sc_ipv4_tcp_max_tw_buckets" ] && log_eval "echo '$sc_ipv4_tcp_max_tw_buckets' > '/proc/sys/net/ipv4/tcp_max_tw_buckets'" [ -n "$sc_ipv4_tcp_max_syn_backlog" ] && log_eval "echo '$sc_ipv4_tcp_max_syn_backlog' > '/proc/sys/net/ipv4/tcp_max_syn_backlog'" [ -n "$sc_ipv4_ip_local_port_range" ] && log_eval "echo '$sc_ipv4_ip_local_port_range' > '/proc/sys/net/ipv4/ip_local_port_range'" +[ -n "$sc_ipv4_tcp_rmem" ] && log_eval "echo '$sc_ipv4_tcp_rmem' > '/proc/sys/net/ipv4/tcp_rmem'" +[ -n "$sc_ipv4_tcp_wmem" ] && log_eval "echo '$sc_ipv4_tcp_wmem' > '/proc/sys/net/ipv4/tcp_wmem'" [ -n "$sc_netdev_max_backlog" ] && log_eval "echo '$sc_netdev_max_backlog' > '/proc/sys/net/core/netdev_max_backlog'" [ -n "$sc_rmem_max" ] && log_eval "echo '$sc_rmem_max' > '/proc/sys/net/core/rmem_max'" [ -n "$sc_wmem_max" ] && log_eval "echo '$sc_wmem_max' > '/proc/sys/net/core/wmem_max'" +[ -n "$sc_sched_features" ] && log_eval "echo '$sc_sched_features' > '/sys/kernel/debug/sched_features'" exit 0 -- 2.23.0

1 0

[PATCH v2 compass-ci] container/manjaro: fix failed to build image in x86 machine
by Liu Yinsi 26 Nov '20

26 Nov '20

[why] when execute manjaro/build in x86_machine, error message: standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error"The command '/bin/sh -c chmod 755 /etc /etc/pacman.d' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/manjaro/Dockerfile | 7 +++++-- container/manjaro/build | 11 ++++++++++- .../root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 .../root/{ => aarch64}/etc/pacman.d/mirrors/China | 0 container/manjaro/root/x86_64/etc/pacman.d/mirrorlist | 1 + .../manjaro/root/x86_64/etc/pacman.d/mirrors/China | 1 + 6 files changed, 17 insertions(+), 3 deletions(-) rename container/manjaro/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) rename container/manjaro/root/{ => aarch64}/etc/pacman.d/mirrors/China (100%) create mode 100644 container/manjaro/root/x86_64/etc/pacman.d/mirrorlist create mode 100644 container/manjaro/root/x86_64/etc/pacman.d/mirrors/China diff --git a/container/manjaro/Dockerfile b/container/manjaro/Dockerfile index 48c5ee6..762acc5 100644 --- a/container/manjaro/Dockerfile +++ b/container/manjaro/Dockerfile @@ -1,11 +1,14 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM manjaroarm/manjaro-aarch64-base +ARG BASE_IMAGE + +FROM ${BASE_IMAGE} MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> +ARG ARCH -COPY root / +COPY root/${ARCH} / RUN chmod 755 /etc /etc/pacman.d RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim diff --git a/container/manjaro/build b/container/manjaro/build index 3ce3049..bc3760d 100755 --- a/container/manjaro/build +++ b/container/manjaro/build @@ -2,4 +2,13 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t manjaro:testbed . +declare -A BASE_IMAGE_DICT + +BASE_IMAGE_DICT=( + ["aarch64"]="manjaroarm/manjaro-aarch64-base" + ["x86_64"]="manjarolinux/base" +) + +BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} + +docker build -t manjaro:testbed --build-arg ARCH="$(arch)" --build-arg BASE_IMAGE="${BASE_IMAGE}" . diff --git a/container/manjaro/root/etc/pacman.d/mirrorlist b/container/manjaro/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/manjaro/root/etc/pacman.d/mirrorlist rename to container/manjaro/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/manjaro/root/etc/pacman.d/mirrors/China b/container/manjaro/root/aarch64/etc/pacman.d/mirrors/China similarity index 100% rename from container/manjaro/root/etc/pacman.d/mirrors/China rename to container/manjaro/root/aarch64/etc/pacman.d/mirrors/China diff --git a/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist b/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..a9def4b --- /dev/null +++ b/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch/ diff --git a/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China b/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China new file mode 100644 index 0000000..a9def4b --- /dev/null +++ b/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China @@ -0,0 +1 @@ +Server = http://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch/ -- 2.23.0

1 0

[PATCH compass-ci 3/5] openeuler: the function executed in docker
by Wang Chenglong 26 Nov '20

26 Nov '20

Install packages, set rootfs, pack images package and other actions in docker. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 40 ++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 container/osimage/openeuler/create-image diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image new file mode 100755 index 0000000..58fe0b1 --- /dev/null +++ b/container/osimage/openeuler/create-image @@ -0,0 +1,40 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +setup_rootfs() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + rm -f /etc/machine-id + rm -f /var/lib/dbus/machine-id + systemd-machine-id-setup +} + + +setup_login() { + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + chmod 644 /tmp/${IMAGE_NAME} +} + +install_packages +setup_rootfs +setup_login +pack_cgz -- 2.23.0

2 2

[PATCH lkp-tests] stats/mysql-server: fix rubocop offenses
by Zhang Yu 26 Nov '20

26 Nov '20

Inspecting 1 file Offenses: mysql-server:18:17: C: Style/RedundantRegexpEscape: Redundant escape inside regexp literal when /\[\s*\d+\%\]/ ^^ Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/mysql-server | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/mysql-server b/stats/mysql-server index d3f3fd79..037d1f0f 100755 --- a/stats/mysql-server +++ b/stats/mysql-server @@ -15,7 +15,7 @@ while (line = STDIN.gets) # 200706 14:35:48 [ 99%] main.subquery_sj_innodb_all w8 [ pass ] 140 # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 - when /\[\s*\d+\%\]/ + when /\[\s*\d+%\]/ results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') -- 2.23.0

1 0

[PATCH compass-ci 5/5] openeuler: the link for one command to build openeuler initramfs image
by Wang Chenglong 26 Nov '20

26 Nov '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- .../openeuler/aarch64/20.03-LTS/build | 30 +++++++++++++++++++ .../initramfs/openeuler/aarch64/20.09/build | 1 + 2 files changed, 31 insertions(+) create mode 100755 rootfs/initramfs/openeuler/aarch64/20.03-LTS/build create mode 120000 rootfs/initramfs/openeuler/aarch64/20.09/build diff --git a/rootfs/initramfs/openeuler/aarch64/20.03-LTS/build b/rootfs/initramfs/openeuler/aarch64/20.03-LTS/build new file mode 100755 index 0000000..016f8f9 --- /dev/null +++ b/rootfs/initramfs/openeuler/aarch64/20.03-LTS/build @@ -0,0 +1,30 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +#. ${CCI_SRC}/container/osimage/centos/lib +#. ${CCI_SRC}/container/defconfig.sh +. $HOME/compass-ci/container/osimage/openeuler/lib +. $HOME/compass-ci/container/defconfig.sh + +export_root_passwd +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -w / + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/files-to-exclude:/tmp/files-to-exclude + -v $DIR/${image}.repo:/etc/yum.repos.d/${image}.repo + -e ROOT_PASSWD=$ROOT_PASSWD + -e IMAGE_NAME=$IMAGE_NAME + $image + /root/bin/create-image +) + +pull_docker_image +"${cmd[@]}" +docker_cp_image +docker_rm init_docker &> /dev/null +echo "build finished" diff --git a/rootfs/initramfs/openeuler/aarch64/20.09/build b/rootfs/initramfs/openeuler/aarch64/20.09/build new file mode 120000 index 0000000..f04acac --- /dev/null +++ b/rootfs/initramfs/openeuler/aarch64/20.09/build @@ -0,0 +1 @@ +../../../../../container/osimage/openeuler/run \ No newline at end of file -- 2.23.0

3 4

[PATCH compass-ci] container/manjaro: fix failed to build image in x86 machine
by Liu Yinsi 26 Nov '20

26 Nov '20

[why] when execute manjaro/build in x86_machine, error message: standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error"The command '/bin/sh -c chmod 755 /etc /etc/pacman.d' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/manjaro/Dockerfile | 7 +++++-- container/manjaro/build | 11 ++++++++++- .../root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 .../root/{ => aarch64}/etc/pacman.d/mirrors/China | 0 container/manjaro/root/x86_64/etc/pacman.d/mirrorlist | 1 + .../manjaro/root/x86_64/etc/pacman.d/mirrors/China | 1 + 6 files changed, 17 insertions(+), 3 deletions(-) rename container/manjaro/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) rename container/manjaro/root/{ => aarch64}/etc/pacman.d/mirrors/China (100%) create mode 100644 container/manjaro/root/x86_64/etc/pacman.d/mirrorlist create mode 100644 container/manjaro/root/x86_64/etc/pacman.d/mirrors/China diff --git a/container/manjaro/Dockerfile b/container/manjaro/Dockerfile index 48c5ee6..762acc5 100644 --- a/container/manjaro/Dockerfile +++ b/container/manjaro/Dockerfile @@ -1,11 +1,14 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM manjaroarm/manjaro-aarch64-base +ARG BASE_IMAGE + +FROM ${BASE_IMAGE} MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> +ARG ARCH -COPY root / +COPY root/${ARCH} / RUN chmod 755 /etc /etc/pacman.d RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim diff --git a/container/manjaro/build b/container/manjaro/build index 3ce3049..114e550 100755 --- a/container/manjaro/build +++ b/container/manjaro/build @@ -2,4 +2,13 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t manjaro:testbed . +declare -A BASE_IMAGE_DICT + +BASE_IMAGE_DICT=( + ["aarch64"]="manjaroarm/manjaro-aarch64-base" + ["x86_64"]="manjarolinux/base" +) + +BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} + +docker build -t manjaro:testbed --build-arg ARCH=$(arch) --build-arg BASE_IMAGE="${BASE_IMAGE}" . diff --git a/container/manjaro/root/etc/pacman.d/mirrorlist b/container/manjaro/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/manjaro/root/etc/pacman.d/mirrorlist rename to container/manjaro/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/manjaro/root/etc/pacman.d/mirrors/China b/container/manjaro/root/aarch64/etc/pacman.d/mirrors/China similarity index 100% rename from container/manjaro/root/etc/pacman.d/mirrors/China rename to container/manjaro/root/aarch64/etc/pacman.d/mirrors/China diff --git a/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist b/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..a9def4b --- /dev/null +++ b/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch/ diff --git a/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China b/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China new file mode 100644 index 0000000..a9def4b --- /dev/null +++ b/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China @@ -0,0 +1 @@ +Server = http://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch/ -- 2.23.0

1 0

[PATCH v3 lkp-tests] spec/submit: add job yaml for multi-args test
by Bai Jing 26 Nov '20

26 Nov '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 9 +++++++ ...unixbench-spec-1-300-whetstone-double.yaml | 25 +++++++++++++++++++ .../unixbench-spec-3-300-shell1.yaml | 25 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 4 files changed, 63 insertions(+) create mode 100644 spec/submit/matrix/unixbench-spec.yaml create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml new file mode 100644 index 00000000..e17322b4 --- /dev/null +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -0,0 +1,9 @@ +suite: unixbench + +runtime: 300s + +nr_task test: + - 1 whetstone-double + - 3 shell1 + +unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml new file mode 100644 index 00000000..9896bcef --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml new file mode 100644 index 00000000..912d10ee --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..a83d2102 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'link matrix' do + submit_job('matrix') + end end -- 2.23.0

2 1

Re: [PATCH compass-ci 3/5] openeuler: the function executed in docker
by Xiao Shenwei 26 Nov '20

26 Nov '20

how about: generate the cgz file in container Reply-To: In-Reply-To: <20201126012754.3857939-3-18509160991(a)163.com> On Thu, Nov 26, 2020 at 09:27:52AM +0800, Wang Chenglong wrote: >Install packages, set rootfs, pack images package and >other actions in docker. > >Signed-off-by: Wang Chenglong <18509160991(a)163.com> >--- > container/osimage/openeuler/create-image | 40 ++++++++++++++++++++++++ > 1 file changed, 40 insertions(+) > create mode 100755 container/osimage/openeuler/create-image > >diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image >new file mode 100755 >index 0000000..58fe0b1 >--- /dev/null >+++ b/container/osimage/openeuler/create-image >@@ -0,0 +1,40 @@ >+#!/bin/bash >+# SPDX-License-Identifier: MulanPSL-2.0+ >+# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. >+ >+install_packages() >+{ >+ yum repolist >+ yum -y install --skip-broken $(</tmp/packages-to-install) >+ yum clean all >+} >+ where use /tmp/file-to-exclude Thanks, Shenwei >+setup_rootfs() >+{ >+ ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime >+ rm -f /etc/machine-id >+ rm -f /var/lib/dbus/machine-id >+ systemd-machine-id-setup >+} >+ >+ >+setup_login() { >+ [ -n "$ROOT_PASSWD" ] || return >+ echo "Changing root password" >+ passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") >+ sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" >+ >+ sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" >+} >+ >+pack_cgz() >+{ >+ echo "Packing package. Please wait." >+ find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} >+ chmod 644 /tmp/${IMAGE_NAME} >+} >+ >+install_packages >+setup_rootfs >+setup_login >+pack_cgz >-- >2.23.0 >

2 1

[PATCH v3 lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..b9b6c8f1 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,18 @@ +#!/usr/bin/env crystal + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + next unless line =~ /transactions:.*\(\s*(\S+)/ + + puts "transactions: #{$1}" +end -- 2.23.0

2 3

[PATCH v3 lkp-tests 2/3] stats/crystal: mysql-server to mysql-server.cr
by Lu Kaiyi 26 Nov '20

26 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/mysql-server.cr | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 stats/crystal/mysql-server.cr diff --git a/stats/crystal/mysql-server.cr b/stats/crystal/mysql-server.cr new file mode 100644 index 00000000..e2ab4b8e --- /dev/null +++ b/stats/crystal/mysql-server.cr @@ -0,0 +1,21 @@ +#!/usr/bin/env crystal +# frozen_string_literal: true + +results = [] of String + +while (line = STDIN.gets) + case line + # The output is as below: + # 200706 14:35:48 [ 99%] main.subquery_sj_innodb_all w8 [ pass ] 140 + # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 + # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 + when /\[\s*\d+\%\]/ + results << line.split(']')[1].split[0] + ".pass: 1" if line.includes?("[ pass ]") + results << line.split(']')[1].split[0] + ".fail: 1" if line.includes?("[ fail ]") + results << line.split(']')[1].split[0] + ".skip: 1" if line.includes?("[ skipped ]") + end +end + +results.each do |item| + puts item +end -- 2.23.0

2 1

[PATCH v3 lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to separate job.yaml, it contains several yaml documents separated by "---" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 19 +++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 27 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 104 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..fbb9dc55 --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,19 @@ +suite: ftq + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..f3e2c553 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: +monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..63bac5cf --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: +monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..a802dc1b --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ce6fed25 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

1 0

[PATCH v4 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to merge job.yaml, it support merging from external file by "<<: job.yaml" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/atomic.yaml | 3 +++ .../merge_yaml/atomic/atomic-defaults.yaml | 19 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 19 +++++++++++++++++++ spec/submit_spec.rb | 4 ++++ 5 files changed, 46 insertions(+) create mode 100644 spec/submit/merge_yaml/atomic.yaml create mode 100644 spec/submit/merge_yaml/atomic/atomic-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/atomic.yaml b/spec/submit/merge_yaml/atomic.yaml new file mode 100644 index 00000000..331cb470 --- /dev/null +++ b/spec/submit/merge_yaml/atomic.yaml @@ -0,0 +1,3 @@ +suite: atomic + +atomic: diff --git a/spec/submit/merge_yaml/atomic/atomic-defaults.yaml b/spec/submit/merge_yaml/atomic/atomic-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/atomic/atomic-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..30019c6c --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: atomic.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/crystal: add libmicro.cr
by Bai Jing 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/crystal/libmicro.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 stats/crystal/libmicro.cr diff --git a/stats/crystal/libmicro.cr b/stats/crystal/libmicro.cr new file mode 100644 index 00000000..2a3f42f0 --- /dev/null +++ b/stats/crystal/libmicro.cr @@ -0,0 +1,10 @@ +#!/usr/bin/env crystal + +# prc thr usecs/call samples errors cnt/samp +# close_tmp 1 1 0.61880 201 0 640 + +while (line = STDIN.gets) + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + + puts "#{$1}: #{$2}" +end -- 2.23.0

1 0

[PATCH lkp-tests] stats/crystal: add libmicro.cr
by Bai Jing 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/crystal/libmicro.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100755 stats/crystal/libmicro.cr diff --git a/stats/crystal/libmicro.cr b/stats/crystal/libmicro.cr new file mode 100755 index 00000000..2a3f42f0 --- /dev/null +++ b/stats/crystal/libmicro.cr @@ -0,0 +1,10 @@ +#!/usr/bin/env crystal + +# prc thr usecs/call samples errors cnt/samp +# close_tmp 1 1 0.61880 201 0 640 + +while (line = STDIN.gets) + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + + puts "#{$1}: #{$2}" +end -- 2.23.0

1 1

[PATCH v3 lkp-tests] lkp-tests/*: update AUR Helper for archlinux
by Wang Yong 26 Nov '20

26 Nov '20

yaourt is unmaintained and yay also support native pacman operation, so only minor changes are needed Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 15 +++++++++++++++ distro/installer/archlinux | 15 +++++++++------ lib/install.sh | 2 +- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/distro/archlinux b/distro/archlinux index 408c5fa23..3f8d62ebc 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -109,3 +109,18 @@ pack_benchmark() ) done } + +add_aur_helper() +{ + echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + pacman -Q yay 2>/dev/null && return + + local user=lkp + local dir=/home/${user} + + mkdir -p ${dir} && chown ${user}:${user} ${dir} + pacman -S --needed --noconfirm git base-devel 2>/dev/null + su - ${user} -c "git clone https://aur.archlinux.org/yay.git" + su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" +} diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 27cdc63cf..3be9e7953 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -2,12 +2,15 @@ # Archlinux package installation -# yaourt is a wrapper for pacman that adds automated access -# to the AUR. -# See: https://wiki.archlinux.org/index.php/Yaourt +# Yet Another Yogurt - An AUR Helper Written in Go +# Yay is based on the design of yaourt, apacman and pacaur +# See: https://github.com/Jguer/yay -# yaourt not support root to build package +# Yay not support root to build package # Use regular user to do this -regular_user=`grep 1000 /etc/passwd | awk -F':' '{ print $1 }'` +. $LKP_SRC/distro/${DISTRO} +add_aur_helper -sudo -u $regular_user yaourt -S --noconfirm $* +regular_user=$(grep 1090 /etc/passwd | awk -F':' '{ print $1 }') + +sudo -u $regular_user yay -S --needed --noconfirm $* diff --git a/lib/install.sh b/lib/install.sh index 881ade9f4..bf7bf21d7 100755 --- a/lib/install.sh +++ b/lib/install.sh @@ -17,7 +17,7 @@ sync_distro_sources() yum update fi ;; redhat) yum update ;; - archlinux) yaourt -Sy ;; + archlinux) yay -Sy ;; opensuse) zypper update ;; oracle) yum update ;; -- 2.23.0

1 0

[PATCH lkp-tests 2/2] spec/submit/link_jobs: update monitors value
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml | 2 +- spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml | 2 +- spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml index 3f26d1ae..9a613c00 100644 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml @@ -26,4 +26,4 @@ pp: freq: 1000 test: add cpufreq_governor: -monitors: {} +monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml index 87945a00..8bf52d0f 100644 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml @@ -26,4 +26,4 @@ pp: freq: 10000 test: add cpufreq_governor: -monitors: {} +monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml index 247cdc37..8cb78287 100644 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml @@ -26,4 +26,4 @@ pp: freq: 20 test: cache cpufreq_governor: -monitors: {} +monitors: -- 2.23.0

1 0

[PATCH lkp-tests 1/2] lib/job.rb: set monitors to nil when no category input
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index c8893433..fe373e86 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -584,8 +584,9 @@ class Job def add_monitors @job[comment_to_symbol('auto generated by add_monitors()')] = nil - @job['monitors'] = {} + @job['monitors'] = nil for_each_in(@job, monitor_params) do |_pk, _h, m_n, _m_args| + @job['monitors'] ||= {} @job['monitors'][m_n] = nil end end -- 2.23.0

1 0

[PATCH compass-ci] doc/job/: add category.md
by Zhang Yu 26 Nov '20

26 Nov '20

Add a document for introducing the category field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/category.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/job/category.md diff --git a/doc/job/category.md b/doc/job/category.md new file mode 100644 index 0000000..9842f58 --- /dev/null +++ b/doc/job/category.md @@ -0,0 +1,21 @@ +# category + +Meaning: +- Every test has its category, the category field is a test type of test cases. +- When you write a job yaml, you can chose one of benchmark, functional, noise-benchmark + to assign the value of category. +- Here are the difference of difference category: + 1. If category: benchmark, default monitors: + kmsg: print the kernel startup information + boot-time: + uptime: start time diafnosis item + iostat: print the time when the data is collected + heartbeat: provide the heartbeat monitors service + ... + other monitors you can see them in $LKP_SRC/include/category/benchmark + 2. If category: functional, default monitors: + kmsg: print the kernel startup information + heartbeat: provide the heartbeat monitors service + meminfo: solid memory usage status or vm usage status + ... + you can see the monitors in $LKP_SRC/include/category/functional -- 2.23.0

2 2

[PATCH lkp-tests] stats/crystal: change the file permissions
by Zhang Yu 26 Nov '20

26 Nov '20

[why] the file permissions should be 664. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mariadb.cr | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 stats/crystal/sysbench-mariadb.cr diff --git a/stats/crystal/sysbench-mariadb.cr b/stats/crystal/sysbench-mariadb.cr old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..1f6709c2 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,17 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + next unless line =~ /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" +end -- 2.23.0

3 3

[PATCH lkp-tests 2/3] iso2rootfs: delete the unnecessary fields
by Yu Chuan 26 Nov '20

26 Nov '20

[Why] 'INITRD_HTTP_HOST' and 'INITRD_HTTP_PORT' should be added by the user on demand at runtime, no need to merge into the git repository. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 -- tests/iso2rootfs | 5 ++--- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 4f3e9c83e6cd..a0ffd280676d 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -7,8 +7,6 @@ iso2rootfs: rootfs_protocol: rootfs_server: rootfs_path: - initrd_http_host: - initrd_http_port: # config rootfs related fields ## supported fields: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 761fbb8b70cf..4a19d42ddd94 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -49,8 +49,6 @@ check_yaml_vars() "rootfs_protocol" "rootfs_server" "rootfs_path" - "initrd_http_host" - "initrd_http_port" ) for yaml_t in "${yaml_vars[@]}" @@ -190,7 +188,8 @@ check_os() get_cgz() { - local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" + + local cgz_url="http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" local cgz_name cgz_name=$(basename "${cgz_url}") local cgz_path_t=${ENV_DIR}/${cgz_name} -- 2.23.0

2 2

[PATCH v2 compass-ci] create define_files in result_root
by Lu Weitao 26 Nov '20

26 Nov '20

[Why] User need debug their result-parse-script which haven't commit in lkp-tests/stats. [How] We already support user write script as "define_file" => {"stats/$script_name" => "$script_content"} in job.yaml, the job.yaml will be save as job to ES DB, write the job['define_file']['stats/$script_name'](value) at $result_root/stats/$script, extract-stats service will parse result by $result_root/stats/$script(user's result-parse-script) firstly Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index bcf6e52..74afee1 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -22,6 +22,8 @@ class StatsWorker if job_id # will consume the job by post-processing job = @es.get_job_content(job_id) result_root = job["result_root"]? + define_files = job["define_files"]? + write_user_stats(result_root.to_s, define_files.as_h) if define_files.is_a?(JSON::Any) result_post_processing(job_id, result_root.to_s, queue_path) @tq.delete_task(queue_path + "/in_process", "#{job_id}") end @@ -139,4 +141,17 @@ class StatsWorker STDERR.puts e.message end end + + def write_user_stats(result_root : String, define_files : Hash(String, JSON::Any)) + return nil unless result_root && File.exists?(result_root) + + define_files.each do |k, v| + if k.match(/^stats/) + Dir.mkdir("#{result_root}/stats") unless File.exists?("#{result_root}/stats") + File.open("#{result_root}/#{k}", "w+", File::Permissions.new(0o775)) do |file| + file.puts v + end + end + end + end end -- 2.23.0

2 2

Re: [PATCH v2 lkp-tests] tats/crystal: add hackbench-git.cr
by Bai Jing 26 Nov '20

26 Nov '20

^^^ stats Thanks, Baijing Reply-To: In-Reply-To: <20201125115826.760733-1-ylzhangah(a)qq.com> On Wed, Nov 25, 2020 at 07:58:26PM +0800, Zhang Yale wrote: >Make ruby2crystal convert > >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > stats/crystal/hackbench-git.cr | 22 ++++++++++++++++++++++ > 1 file changed, 22 insertions(+) > create mode 100644 stats/crystal/hackbench-git.cr >diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr >new file mode 100644 >index 00000000..2c043fc9 >--- /dev/null >+++ b/stats/crystal/hackbench-git.cr >@@ -0,0 +1,22 @@ >+#!/usr/bin/env crystal >+ >+# --------socket thread num=20-------- >+# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 >+# Running with 20*40 (== 800) tasks. >+# Time: 0.495 >+# ... >+ >+time = [] of Float64 >+ >+while (line = gets) >+ case line >+ when /^-+(\w+ \w+ \w+)/ >+ args = $1.tr(" ", "_") >+ when /^Running with .* $== (\d+)$ tasks/ >+ tasks = $1.to_i >+ when /^Time:/ >+ time << line.split[1].to_f >+ end >+end >+ >+puts "#{args}_#{tasks}: #{time.sum / time.size}" >-- >2.23.0 >

2 1

[PATCH v3 lkp-tests 3/3] tests/iozone: refactor format optimization
by Lu Kaiyi 26 Nov '20

26 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/iozone b/tests/iozone index 5186766f..923d8331 100755 --- a/tests/iozone +++ b/tests/iozone @@ -47,8 +47,7 @@ if [ -n "$block_size" ]; then ;; "preadv") args+=" -i 12" ;; - *) echo "invalid test option, please check again!" - exit 1 + *) echo "invalid test option, please check again!" && exit 1 ;; esac done -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/3] stats/mysql-server: remove redundant whitespace
by Lu Kaiyi 26 Nov '20

26 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/mysql-server | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/stats/mysql-server b/stats/mysql-server index d3f3fd79..776f3f2e 100755 --- a/stats/mysql-server +++ b/stats/mysql-server @@ -16,9 +16,9 @@ while (line = STDIN.gets) # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 when /\[\s*\d+\%\]/ - results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') - results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') - results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') + results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') + results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') + results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tats/crystal: add hackbench-git.cr
by Zhang Yale 25 Nov '20

25 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/crystal/hackbench-git.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 stats/crystal/hackbench-git.cr diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr new file mode 100644 index 00000000..2c043fc9 --- /dev/null +++ b/stats/crystal/hackbench-git.cr @@ -0,0 +1,22 @@ +#!/usr/bin/env crystal + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +time = [] of Float64 + +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(" ", "_") + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + time << line.split[1].to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 0

[PATCH lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 25 Nov '20

25 Nov '20

[why] Make ruby2crystal convert. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..1bab78c0 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,19 @@ +#!/usr/bin/env crystal + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + case line + when /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..c604bfa --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,27 @@ +# runtime + +Meaning: +- Seconds to run before exiting. +- Limit the test's run time in $runtime. + +Usage example: + +```bash +#!/bin/sh +# - runtime +``` + +`runtime` is a parameter of test script. +For any test case needing parameter(s), +need such comment line and follow exactly +the same format `# - parameter`. + +```yaml +runtime: 300s +``` + +```yaml +runtime: 1d +sleep: +``` + -- 2.23.0

1 0

[PATCH lkp-tests 3/3] iso2rootfs: add comment to iso2rootfs.yaml
by Yu Chuan 25 Nov '20

25 Nov '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index a0ffd280676d..2c4d3a93ff62 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -1,9 +1,18 @@ suite: iso2rootfs category: benchmark iso2rootfs: + # iso related fields to be used to generate rootfs iso_os: iso_arch: iso_version: + + # place the result rootfs related fields + ## 1. result rootfs will be placed in the following localtion on the + ## remote file server: + ## - {remote_file_server}/{rootfs_path}/{iso_os}/{iso_arch}/ + ## 2. remote file server protocols current supported: + ## - nfs + ## - cifs rootfs_protocol: rootfs_server: rootfs_path: -- 2.23.0

1 0

[PATCH lkp-tests 1/3] iso2rootfs: delete the meaningless field
by Yu Chuan 25 Nov '20

25 Nov '20

[Why] The 'iso2rootfs_cgz' field will always be 'iso2rootfs.cgz', so delete this field, and use the 'iso2rootfs.cgz' directly. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 1 - tests/iso2rootfs | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 1a643b538ac1..4f3e9c83e6cd 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -1,7 +1,6 @@ suite: iso2rootfs category: benchmark iso2rootfs: - iso2rootfs_cgz: iso_os: iso_arch: iso_version: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index fcaadaa1ba46..761fbb8b70cf 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -43,7 +43,6 @@ check_yaml_vars() "os" "os_arch" "os_version" - "iso2rootfs_cgz" "iso_os" "iso_arch" "iso_version" @@ -191,7 +190,7 @@ check_os() get_cgz() { - local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/${iso2rootfs_cgz}" + local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" local cgz_name cgz_name=$(basename "${cgz_url}") local cgz_path_t=${ENV_DIR}/${cgz_name} -- 2.23.0

1 0

[PATCH v5 lkp-tests] job: fix each_jobs to fit multi-args
by Bai Jing 25 Nov '20

25 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h2 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} Because "@job2=hash" will merge the h2 to @job after "each_job". We need to merge the h2 to @job before "each_job". So do the merge advance and set @job2={} Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index c8893433..9272392e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -730,7 +730,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..296fc6e --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,20 @@ +# runtime + +Meaning: +Limit run time to runtime seconds. + +Usage example: + +```#/bin/bash +# - runtime +``` + +`runtime` is a parameter of test script. For any test case +needing paramter(s), need such comment line and follow exactly +the same format `# - parameter`. + +job.yaml: + +``` +runtime: 300s +``` -- 2.23.0

2 2