- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to separate job.yaml, it contains several yaml documents separated by "---" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 19 +++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 27 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 104 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..fbb9dc55 --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,19 @@ +suite: ftq + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..f3e2c553 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: +monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..63bac5cf --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: +monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..a802dc1b --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index a83d2102..169d7812 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -44,4 +44,8 @@ describe 'submit job spec' do it 'link matrix' do submit_job('matrix') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

1 0

[PATCH v4 lkp-tests] setup/sysctl: add system parameters
by Zhang Yu 26 Nov '20

26 Nov '20

[why] When testing need to setup system parameters. [how] Add parameters to setup/sysctl. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- setup/sysctl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/sysctl b/setup/sysctl index 81374939..2ace4db2 100755 --- a/setup/sysctl +++ b/setup/sysctl @@ -202,6 +202,9 @@ # - sc_netdev_max_backlog # - sc_rmem_max # - sc_wmem_max +# - sc_ipv4_tcp_rmem +# - sc_ipv4_tcp_wmem +# - sc_sched_features . $LKP_SRC/lib/reproduce-log.sh @@ -405,8 +408,11 @@ [ -n "$sc_ipv4_tcp_max_tw_buckets" ] && log_eval "echo '$sc_ipv4_tcp_max_tw_buckets' > '/proc/sys/net/ipv4/tcp_max_tw_buckets'" [ -n "$sc_ipv4_tcp_max_syn_backlog" ] && log_eval "echo '$sc_ipv4_tcp_max_syn_backlog' > '/proc/sys/net/ipv4/tcp_max_syn_backlog'" [ -n "$sc_ipv4_ip_local_port_range" ] && log_eval "echo '$sc_ipv4_ip_local_port_range' > '/proc/sys/net/ipv4/ip_local_port_range'" +[ -n "$sc_ipv4_tcp_rmem" ] && log_eval "echo '$sc_ipv4_tcp_rmem' > '/proc/sys/net/ipv4/tcp_rmem'" +[ -n "$sc_ipv4_tcp_wmem" ] && log_eval "echo '$sc_ipv4_tcp_wmem' > '/proc/sys/net/ipv4/tcp_wmem'" [ -n "$sc_netdev_max_backlog" ] && log_eval "echo '$sc_netdev_max_backlog' > '/proc/sys/net/core/netdev_max_backlog'" [ -n "$sc_rmem_max" ] && log_eval "echo '$sc_rmem_max' > '/proc/sys/net/core/rmem_max'" [ -n "$sc_wmem_max" ] && log_eval "echo '$sc_wmem_max' > '/proc/sys/net/core/wmem_max'" +[ -n "$sc_sched_features" ] && log_eval "echo '$sc_sched_features' > '/sys/kernel/debug/sched_features'" exit 0 -- 2.23.0

1 0

[PATCH v3 compass-ci] web_backend.rb: fix : compare_groups_matrices needs arg suites_list
by Li Yuanchao 26 Nov '20

26 Nov '20

compare_groups_matrices needs three args, but now only get two, lost suites_list. divide method to pass rubocop limit Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..90c8a58 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -147,6 +147,20 @@ def get_es_must(params) must end +def _get_group_matrices(query, dimension) + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + while matrices.empty? + from += size + break if from > total + + query[:from] = from + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + end + [matrices, suites_list] +end + def get_groups_matrices(conditions, dimension, must, size, from) must += build_mutli_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } @@ -166,27 +180,17 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - while matrices.empty? - from += size - break if from > total - - query[:from] = from - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - end - matrices + matrices, suites_list = _get_groups_matrices(query, dimension) end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: about multi-device test
by Ren Wen 26 Nov '20

26 Nov '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- doc/multi-device-test.md | 109 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 doc/multi-device-test.md diff --git a/doc/multi-device-test.md b/doc/multi-device-test.md new file mode 100644 index 0000000..1be8c79 --- /dev/null +++ b/doc/multi-device-test.md @@ -0,0 +1,109 @@ +# Multi-device Test + +在进行需要多台机器的测试的时候, 需要搭建多台机器的环境. + +## When using multi-device + + * 当被测试程序需要启动一个守护进程时 + * 当执行测试时需要访问另外的机器时 + +## How to run a multi-device test + +在要提交的job.yaml中, 需要一个**cluster**字段, 它的值是一个多机环境的配 +置文件的文件名, 配置文件的内容可以参考: *lab-z9/cluster/*. +调度器会根据配置文件的内容将该job拆分为对应机器数量的job, 然后分别提交到 +相应的队列中. + +那么如何对指定的机器执行指定的测试呢? +在job.yaml中增加这样的内容: + +``` yaml +if role <role01> + <test_script01>: + <test_script02>: + +if role <role02> + <test_script03>: + <test_script04>: + +... +``` + +在多机的配置文件中, 会给每台机器赋予一个角色. 当每台机器取得自己队列中的 +job时, 按照自己的角色去执行相应角色所对应的脚本. + +比如说: 一台多机测试中的机器的角色为*role01*, 那么它会去寻找脚本 +*test_script01*, *test_script02*, 然后去执行. + +### how to find test script + +在lkp-tests的**daemon**, **tests**目录下查找这些脚本. +假设在**daemon**下找到了*test_script01*, 则将该脚本作为`start_daemon`的参 +数运行. 如果在**tests**下找到了*test_script01*, 则作为`run_test`的参数去 +运行. 如果没有找到相应的文件, 则按变量导出. + +### about `start_daemon` and `run_test` + +用于启动脚本的这2个函数位于**lkp-tests/lib/job.sh**. +要搞清楚多机的流程, 就得明白这两个函数是怎么去做的. + +#### the processes of `start_damon` + + 1. 运行在**daemon**文件夹下找到的脚本. 如果运行成功, 向调度器报告一个状态: + `started`, 调度器记录该机器的状态为: `start`; 执行失败, 则报告: + `failed`, 调度器记录为: `abort`, 然后退出执行. 状态可以查看redis的 + key: `HGETALL sched/cluster_state`. + 2. 向调度器报告一个状态: `write_state`, 并携带一些机器相关的参数: + * `node_roles`: 机器的角色 + * `ip`: 通过命令`hostname -I`取得的第一个值 + * `direct_macs`: 多机配置文件中该机器的**macs**字段 + * `direct_ips`: 调度器根据多机配置文件中的**ip0**字段配置的直连ip, 与 + 配置文件中**macs**的值一一对应 + 此时调度器将这些参数记录为每台机器相关的信息, 可以查看redis的key: + **sched/cluster_state**. + 3. 将**direct_ips**的值一一绑定到机器中的**direct_macs**的相应设备上. + 4. 向调度器报告一个状态: `wait_ready`, 调度器将其记录为: `ready`; 与此 + 同时, 调度器查看是否多机测试中的所有机器都准备好了. 如果每一台机器 + 都准备好了, 则返回`ready`; 如果其中一台的状态为`abort`, 则返回 + `abort`, 然后退出执行; 其余情况, 则循环查询, 直到前2种情形发生. + 5. 向调度器报告一个状态: `roles_ip`, 并将调度器的返回值导出为环境变量. + 比如当返回值为: `ip=ip01`时, 执行: `export ip=ip01`. + 6. 向调度器报告一个状态: `wait_finish`, 调度器将其记录为: `finish`; 与 + 此同时, 调度器查看是否多机测试中的所有机器的状态都为`finish`. 如果 + 都是`finish`, 则返回`finish`; 如果其中一台的状态为`abort`, 则返回 + `abort`, 然后退出执行; 其余情况, 则循环查询, 直到前2种情形发生. + 7. 执行完成. + +#### the processes of `run_test` + +步骤1,2,3,4同上述2,3,4,5. + + 1. 向调度器报告一个状态: `write_state`, 并携带一些机器相关的参数: + * `node_roles`: 机器的角色 + * `ip`: 通过命令`hostname -I`取得的第一个值 + * `direct_macs`: 多机配置文件中该机器的**macs**字段 + * `direct_ips`: 调度器根据多机配置文件中的**ip0**字段配置的直连ip, 与 + 配置文件中**macs**的值一一对应 + 此时调度器将这些参数记录为每台机器相关的信息, 可以查看redis的key: + **sched/cluster_state**. + 2. 将**direct_ips**的值一一绑定到机器中的**direct_macs**的相应设备上. + 3. 向调度器报告一个状态: `wait_ready`, 调度器将其记录为: `ready`; 与此 + 同时, 调度器查看是否多机测试中的所有机器都准备好了. 如果每一台机器 + 都准备好了, 则返回`ready`; 如果其中一台的状态为`abort`, 则返回 + `abort`, 然后退出执行; 其余情况, 则循环查询, 直到前2种情形发生. + 4. 向调度器报告一个状态: `roles_ip`, 并将调度器的返回值导出为环境变量. + 比如当返回值为: `ip=ip01`时, 执行: `export ip=ip01` + 5. 运行在**tests**文件夹下找到的脚本. 运行完成, 向调度器报告一个状态: + `finished`, 调度器记录为: `finish`; 执行失败, 则报告: `failed`, 调 + 度器记录为: `abort`, 然后退出执行. + 6. 执行完成. + +#### others + +`start_daemon`和`run_test`还会报告一个`abort`状态: 循环100次向调度器发 +起请求, 查看所有机器是否全部`ready`或`finish`, 如果循环结束仍然返回的是 +`retry`, 则报告该状态, 并结束执行. + +当调度器接受到与之相关的请求时, 会阻塞(或循环)查询这些机器的状态, 因此只 +需返回`ready`(`finish`)或`abort`. 所以不会触发循环100次请求这个过程, 因 +此不会报告`abort`这个状态. -- 2.23.0

1 0

[PATCH lkp-tests] distro/*: add exit when failed download packages
by Wang Yong 26 Nov '20

26 Nov '20

now when download packages failed, echo error message and $? is 0, so add exit 1 to get last_state Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 1 + distro/archlinux | 1 + distro/debian | 1 + 3 files changed, 3 insertions(+) diff --git a/distro/aliyun b/distro/aliyun index 79656f0e4..598243d8e 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -126,6 +126,7 @@ pack_benchmark_deps() yum_pack else echo "failed to pack-deps $BM_NAME" >&2 + exit 1 fi done } diff --git a/distro/archlinux b/distro/archlinux index 3f8d62ebc..3942080a6 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -24,6 +24,7 @@ pack_benchmark_deps() pacman_pack else echo "failed to pack-deps $BM_NAME" >&2 + exit 1 fi done } diff --git a/distro/debian b/distro/debian index 13c4e983a..8821e3ddd 100755 --- a/distro/debian +++ b/distro/debian @@ -271,6 +271,7 @@ pack_benchmark_deps() echo "$PACKAGE_LIST" > $pack_to/.${BM_NAME}.packages else echo "failed to pack-deps $BM_NAME" >&2 + exit 1 fi rm -fr "$work_dir" -- 2.23.0

1 0

[PATCH compass-ci 1/4] answerback-email: add new options
by Luan Shengde 26 Nov '20

26 Nov '20

add option for answerback-email tool -n|--name name specify my_name for user when assigning account -l|--login Y|y|N|n enable/disable user login permission -u|--update enable update configuration for assigned account -u option enable update for: my_name my_ssh_pubkey generate sshkey on jumper reset user's login permission Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 147 +++++++++++++++---- 1 file changed, 120 insertions(+), 27 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 5ddc451..bffba17 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -14,6 +14,7 @@ require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' require_relative 'build-send-account-email' +require_relative 'build-update-email' names = Set.new %w[ JUMPER_HOST @@ -33,25 +34,61 @@ my_info = { 'my_email' => nil, 'my_name' => nil, 'my_commit_url' => nil, - 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => [], - 'gen_sshkey' => false + 'my_uuid' => nil, + 'my_login_name' => nil, + 'my_ssh_pubkey' => [] } -def init_info(mail_content, my_info) +# stdin_info is used to store infos added with option: +# -e email_address +# -n name +# -s pubkey_file +# email_info is used to store infos parsed from email_file with option: +# -f email_file +# my_info_es is used to store info read from ES when update user +# when use --update option +# stdin_info, email_info, my_info_es have different priority +# stdin_info > email_info > my_info_es +# when assigning account or update conf for account +# if they have the same key, my_info will use the value with higher priority. +# conf_info is used to store keys used to config the account +stdin_info = {} +email_info = {} +my_info_es = {} +conf_info = { + 'gen_sshkey' => false, + 'enable_login' => true, + 'is_update_account' => false +} + +def init_info(mail_content, email_info, my_info) my_info['my_email'] = mail_content.from[0] - my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + email_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') return if mail_content.attachments.empty? - my_info['my_ssh_pubkey'] << mail_content.attachments[0].body.decoded + email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip +end + +def read_my_login_name(my_email, my_info_es) + my_account_info_str = %x(curl -XGET localhost:9200/accounts/_doc/#{my_email}) + my_account_info = YAML.safe_load my_account_info_str + message = "No such email found from the ES: #{my_email}" + raise message unless my_account_info['found'] + + my_info_es.update my_account_info['_source'] end options = OptionParser.new do |opts| - opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey]\n" - opts.banner += " -e or -f is required\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|Y|n|N] [--update]\n" + opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' + opts.banner += "[-g|--gen-sshkey] [--login y|Y|n|N] [--update]\n" + opts.banner += " -e|-f is required when applying account or updating account\n" + opts.banner += " -n is require when assigning account with -e\n" opts.banner += " -s is optional when use -e\n" - opts.banner += ' -g is optional, used to generate sshkey for user' + opts.banner += " -g is optional, used to generate sshkey for user\n" + opts.banner += " -u is require when updating an account\n" + opts.banner += ' -l is optional, used to enable/disable login permission' opts.separator '' opts.separator 'options:' @@ -60,18 +97,37 @@ options = OptionParser.new do |opts| my_info['my_email'] = email_address end + opts.on('-n name', '--name name', 'appoint name') do |name| + stdin_info['my_name'] = name + end + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp + stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| mail_content = Mail.read(email_file) - init_info(mail_content, my_info) + init_info(mail_content, email_info, my_info) end opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do - my_info['gen_sshkey'] = true + conf_info['gen_sshkey'] = true + end + + opts.on('-u', '--update', 'updata user account') do + read_my_login_name(my_info['my_email'], my_info_es) + conf_info['is_update_account'] = true + end + + opts.on('-l Y|y|N|n', '--login Y|y|N|n', 'enable/disable user login permission') do |value| + if %w[y Y].include? value + conf_info['enable_login'] = true + elsif %w[n N].include? value + conf_info['enable_login'] = false + else + raise 'invalid parameter, please use y|Y|n|N' + end end opts.on_tail('-h', '--help', 'show this message') do @@ -82,33 +138,70 @@ end options.parse!(ARGV) -def apply_account(my_info) - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') +def apply_account(my_info, conf_info) + apply_info = {} + apply_info.update my_info + apply_info.update conf_info + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account(my_info) +def check_my_email(my_info) message = "No email address specified\n" - message += "use -e to add a email address\n" + message += "use -e to add a email address for applying account\n" message += 'or use -f to add a email file' - raise message if my_info['my_email'].nil? - account_info = apply_account(my_info) - my_info['my_login_name'] = account_info['my_login_name'] + raise message if my_info['my_email'].empty? +end +def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) + new_email_pubkey = email_info.delete 'new_email_pubkey' + new_stdin_pubkey = stdin_info.delete 'new_ssh_pubkey' + new_pubkey = new_stdin_pubkey || new_email_pubkey + + my_info.update my_info_es unless my_info_es.empty? + my_info.update email_info unless email_info.empty? + my_info.update stdin_info unless stdin_info.empty? + + return if new_pubkey.nil? + return if my_info['my_ssh_pubkey'].include? new_pubkey + + my_info['my_ssh_pubkey'].insert(0, new_pubkey) +end + +def build_my_info_from_account_info(my_info, account_info, conf_info) unless account_info['my_jumper_pubkey'].nil? - my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + return if my_info['my_ssh_pubkey'][-1] == account_info['my_jumper_pubkey'] + + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] end - my_info.delete 'gen_sshkey' + my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] +end + +def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) + check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] + build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) + + message = 'No my_name found, please use -n to add one' + raise message if my_info['my_name'].nil? + + account_info = apply_account(my_info, conf_info) + build_my_info_from_account_info(my_info, account_info, conf_info) + store_account_info(my_info) - send_mail(my_info, account_info) + send_mail(my_info, account_info, conf_info) end -def send_mail(my_info, account_info) - message = build_message(my_info['my_email'], account_info) - +def send_mail(my_info, account_info, conf_info) + message = if conf_info['is_update_account'] + build_update_message(my_info['my_email'], account_info, conf_info) + else + build_message(my_info['my_email'], account_info) + end %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end @@ -117,4 +210,4 @@ def store_account_info(my_info) es.put_source_by_id(my_info['my_email'], my_info) end -send_account(my_info) +send_account(my_info, conf_info, email_info, my_info_es, stdin_info) -- 2.23.0

2 2

[PATCH compass-ci] web_backend.rb: fix : compare_groups_matrices needs arg suites_list
by Li Yuanchao 26 Nov '20

26 Nov '20

compare_groups_matrices needs three args, but now only get two, lost suites_list. divide method to pass rubocop limit Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..b5b06b4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -147,6 +147,20 @@ def get_es_must(params) must end +def _get_group_matrices(query, dimension) + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + while matrices.empty? + from += size + break if from > total + + query[:from] = from + result = es_query(query) + matrices, suites_list = combine_group_query_data(result, dimension) + end + [matrices, suites_list] +end + def get_groups_matrices(conditions, dimension, must, size, from) must += build_mutli_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } @@ -166,27 +180,18 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - while matrices.empty? - from += size - break if from > total - - query[:from] = from - result = es_query(query) - matrices = combine_group_query_data(result, dimension) - end - matrices + matrices, suites_list = _get_groups_matrices(query, dimension) + [matrices, suites_list] end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH v3 lkp-tests] submit: fix the processes of option: --my-queue
by Ren Wen 26 Nov '20

26 Nov '20

If use options '--my-queue' and '-o' at the same time, the output job file will not contain the value of the covered field 'queue'. So pass the 'jobs' not 'job_hash'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index be588bed..23518eb0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -112,6 +112,7 @@ ARGV.each do |jobfile| jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') + add_my_queue(jobs) if opt_my_queue jobs['node_roles'] ||= 'server client' if jobs['cluster'] jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] @@ -144,7 +145,6 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) - add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/job: add category.md
by Zhang Yu 26 Nov '20

26 Nov '20

Add a document for introducing the category field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/category.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/job/category.md diff --git a/doc/job/category.md b/doc/job/category.md new file mode 100644 index 0000000..6425318 --- /dev/null +++ b/doc/job/category.md @@ -0,0 +1,21 @@ +# category + +Meaning: +- Each test has its category, the category field is a test type of test cases. +- When you write a job yaml, you can choose one of benchmark, functional, noise-benchmark + to assign the value of category. +- Here are some differences about different category values: + 1. If category is benchmark, it will monitor some system information: + kmsg: print the kernel startup information + boot-time: start time diagnosis item + uptime: display how long the system has been running + iostat: print the time when the data is collected + heartbeat: provide the heartbeat monitors service + ... + other monitors you can see them in $LKP_SRC/include/category/benchmark + 2. If category is functional, it will monitor some system information: + kmsg: print the kernel startup information + heartbeat: provide the heartbeat monitors service + meminfo: solid memory usage status or vm usage status + ... + you can see the monitors in $LKP_SRC/include/category/functional -- 2.23.0

1 0

[PATCH v2 lkp-tests] submit: fix the processes of option: --my-queue
by Ren Wen 26 Nov '20

26 Nov '20

If use options '--my-queue' and '-o' at the same time, the output job file will not contain the value of the covered field 'queue'. So pass the 'jobs' not 'job_hash'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index be588bed..9510d53d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -108,6 +108,7 @@ ARGV.each do |jobfile| jobs = Job2sh.new jobs.cmdline_defaults = opt_cmdline_defaults jobs.overrides = opt_set_key_value + add_my_queue(jobs) if opt_my_queue jobs.load(jobfile, true) || next jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] @@ -144,7 +145,6 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) - add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

2 2

[PATCH compass-ci] doc/job/: add lkp_initrd_user.md
by Zhang Yale 26 Nov '20

26 Nov '20

Add a document for introducing the lkp_initrd_user field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/lkp_initrd_user.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/job/lkp_initrd_user.md diff --git a/doc/job/lkp_initrd_user.md b/doc/job/lkp_initrd_user.md new file mode 100644 index 0000000..b4a0982 --- /dev/null +++ b/doc/job/lkp_initrd_user.md @@ -0,0 +1,10 @@ +# lkp_inird_user + +Meaning: +- "lkp_initrd_user" is one of the init_field, the default value is "latest". +- User can set lkp_initrd_user to $USER or the default value in "/home/$USER/.config/compass-ci/default/$USER.yaml. +- If the user's lkp_tests code is changed, build & run the lkp_initrd container which in "${CCI_SRC}/container/lkp-initrd" + to pack a new lkp-${os_arch}.cgz. +- The new lkp-${os_arch}.cgz file is generated in the "${SRV_INITRD}/lkp/${lkp_initrd_user}/lkp-${os_arch}.cgz". +- When user submit one job, qemu-pxe or other testbox start mount the <"lkp"> initrd to consume the user submited <"job"> + "initrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/lkp/${lkp_initrd_user}/lkp-${os_arch}.cgz". -- 2.23.0

3 5

[PATCH compass-ci 1/5] openeuler: configure files about build image with one comman
by Wang Chenglong 26 Nov '20

26 Nov '20

[Contains] openeule-${os_version}.repo: yum configuration files files-to-exclude: list of files that need to be excluded packages-to-install: list of packages that need to install README.md: descriptive documentation Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/README.md | 15 +++++++ container/osimage/openeuler/files-to-exclude | 11 +++++ .../openeuler/openeuler-20.03-lts.repo | 43 +++++++++++++++++++ .../osimage/openeuler/openeuler-20.09.repo | 43 +++++++++++++++++++ .../osimage/openeuler/packages-to-install | 10 +++++ 5 files changed, 122 insertions(+) create mode 100644 container/osimage/openeuler/README.md create mode 100644 container/osimage/openeuler/files-to-exclude create mode 100644 container/osimage/openeuler/openeuler-20.03-lts.repo create mode 100644 container/osimage/openeuler/openeuler-20.09.repo create mode 100644 container/osimage/openeuler/packages-to-install diff --git a/container/osimage/openeuler/README.md b/container/osimage/openeuler/README.md new file mode 100644 index 0000000..cea68f9 --- /dev/null +++ b/container/osimage/openeuler/README.md @@ -0,0 +1,15 @@ +# Use the tool to create a new openeuler-${os_version} initramfs image. + +Usage: + cd ${CCI_SRC}/rootfs/initramfs/openeuler/aarch64/${os_version}/ + ./build + +Some configuration items: +./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + +./files-to-exclude + If you want remove some unnecessary files, you can write the names in ./files-to-exclude + +$HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. diff --git a/container/osimage/openeuler/files-to-exclude b/container/osimage/openeuler/files-to-exclude new file mode 100644 index 0000000..3854947 --- /dev/null +++ b/container/osimage/openeuler/files-to-exclude @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/files-to-exclude +/tmp/packages-to-install diff --git a/container/osimage/openeuler/openeuler-20.03-lts.repo b/container/osimage/openeuler/openeuler-20.03-lts.repo new file mode 100644 index 0000000..691f438 --- /dev/null +++ b/container/osimage/openeuler/openeuler-20.03-lts.repo @@ -0,0 +1,43 @@ +#generic-repos is licensed under the Mulan PSL v2. +#You can use this software according to the terms and conditions of the Mulan PSL v2. +#You may obtain a copy of Mulan PSL v2 at: +# http://license.coscl.org.cn/MulanPSL2 +#THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR +#IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR +#PURPOSE. +#See the Mulan PSL v2 for more details. + +[OS] +name=OS +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/OS/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/OS/$basearch/RPM-GPG-KEY-openEuler + +[everything] +name=everything +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/everything/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/everything/$basearch/RPM-GPG-KEY-openEuler + +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/EPOL/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/EPOL/$basearch/RPM-GPG-KEY-openEuler + +[debuginfo] +name=debuginfo +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/debuginfo/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/debuginfo/$basearch/RPM-GPG-KEY-openEuler + +[source] +name=source +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/source/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/source/RPM-GPG-KEY-openEuler diff --git a/container/osimage/openeuler/openeuler-20.09.repo b/container/osimage/openeuler/openeuler-20.09.repo new file mode 100644 index 0000000..1da7571 --- /dev/null +++ b/container/osimage/openeuler/openeuler-20.09.repo @@ -0,0 +1,43 @@ +#generic-repos is licensed under the Mulan PSL v2. +#You can use this software according to the terms and conditions of the Mulan PSL v2. +#You may obtain a copy of Mulan PSL v2 at: +# http://license.coscl.org.cn/MulanPSL2 +#THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR +#IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR +#PURPOSE. +#See the Mulan PSL v2 for more details. + +[OS] +name=OS +baseurl=http://repo.openeuler.org/openEuler-20.09/OS/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/OS/$basearch/RPM-GPG-KEY-openEuler + +[everything] +name=everything +baseurl=http://repo.openeuler.org/openEuler-20.09/everything/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/everything/$basearch/RPM-GPG-KEY-openEuler + +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/openEuler-20.09/EPOL/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/EPOL/$basearch/RPM-GPG-KEY-openEuler + +[debuginfo] +name=debuginfo +baseurl=http://repo.openeuler.org/openEuler-20.09/debuginfo/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/debuginfo/$basearch/RPM-GPG-KEY-openEuler + +[source] +name=source +baseurl=http://repo.openeuler.org/openEuler-20.09/source/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/source/RPM-GPG-KEY-openEuler diff --git a/container/osimage/openeuler/packages-to-install b/container/osimage/openeuler/packages-to-install new file mode 100644 index 0000000..f17eee1 --- /dev/null +++ b/container/osimage/openeuler/packages-to-install @@ -0,0 +1,10 @@ +cpio +systemd-udev.aarch64 +cifs-utils +nfs-utils +hostname +curl +openssh-server +openssl +iputils +wget -- 2.23.0

3 4

[PATCH v1 lkp-tests] spec/submit: add job yaml for multi-args test
by Bai Jing 26 Nov '20

26 Nov '20

[test cmd] respec $LKP_SRC/spec/submit_spec.rb [output] lkp-tests/spec/submit/matrices-args/unixbench-spec.yaml => /lkp-tests/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml lkp-tests/spec/submit/matrices-args/unixbench-spec.yaml => /lkp-tests/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrices-args/unixbench-spec.yaml | 8 ++++++ ...unixbench-spec-1-300-whetstone-double.yaml | 25 +++++++++++++++++++ .../unixbench-spec-3-300-shell1.yaml | 25 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 4 files changed, 62 insertions(+) create mode 100644 spec/submit/matrices-args/unixbench-spec.yaml create mode 100644 spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml diff --git a/spec/submit/matrices-args/unixbench-spec.yaml b/spec/submit/matrices-args/unixbench-spec.yaml new file mode 100644 index 00000000..a763927b --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec.yaml @@ -0,0 +1,8 @@ +suite: unixbench + +runtime: 300s + +nr_task test: + - 1 whetstone-double + - 3 shell1 +unixbench: diff --git a/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml new file mode 100644 index 00000000..16af43d2 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: +monitors: {} diff --git a/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml new file mode 100644 index 00000000..5c0526a1 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: +monitors: {} diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..3a9b0865 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'link matrices-args' do + submit_job('matrices-args') + end end -- 2.23.0

2 2

[PATCH lkp-tests] tats/crystal: add hackbench-git.cr
by Zhang Yale 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/crystal/hackbench-git.cr | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 stats/crystal/hackbench-git.cr diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr new file mode 100644 index 00000000..214dcd02 --- /dev/null +++ b/stats/crystal/hackbench-git.cr @@ -0,0 +1,23 @@ +#!/usr/bin/env crystal + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +time = [] of Float64 + +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(" ", "_") + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

2 2

[PATCH compass-ci 4/5] openeuler: build openeuler initramfs image with one command
by Wang Chenglong 26 Nov '20

26 Nov '20

[Why] Use the tool to create a new openeuler-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./packages-to-install. [How] We create a link button for the user to use. $CCI_SRC/rootfs/initramfs/openeuler/aarch64/${os_version}/build Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/run | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100755 container/osimage/openeuler/run diff --git a/container/osimage/openeuler/run b/container/osimage/openeuler/run new file mode 100755 index 0000000..c92c59f --- /dev/null +++ b/container/osimage/openeuler/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +export_root_passwd +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/files-to-exclude:/tmp/files-to-exclude + -v $DIR/${image}.repo:/etc/yum.repos.d/openEuler.repo + -e ROOT_PASSWD=$ROOT_PASSWD + -e IMAGE_NAME=$IMAGE_NAME + $image + /root/bin/create-image +) + +pull_docker_image +"${cmd[@]}" +docker_cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

3 6

[PATCH compass-ci 2/2] mail-robot: add sched info
by Luan Shengde 26 Nov '20

26 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/mail-robot/start b/container/mail-robot/start index 5827e18..1c55dd3 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -12,6 +12,8 @@ names = Set.new %w[ JUMPER_HOST JUMPER_PORT SEND_MAIL_PORT + SCHED_HTTPS_HOST + SCHED_HTTPS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: add sched info in email message
by Luan Shengde 26 Nov '20

26 Nov '20

for user to add the sched info in the default yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b793232..4754c4c 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -25,6 +25,8 @@ def build_apply_account_email(my_info) my_email: #{my_info['my_email']} my_name: #{my_info['my_name']} my_uuid: #{my_info['my_uuid']} + SCHED_HTTPS_HOST: #{ENV['SCHED_HTTPS_HOST']} + SCHED_HTTPS_PORT: #{ENV['SCHED_HTTPS_PORT']} EOF 2) download lkp-tests and dependencies -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job/: add category.md
by Zhang Yu 26 Nov '20

26 Nov '20

Add a document for introducing the category field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/category.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/job/category.md diff --git a/doc/job/category.md b/doc/job/category.md new file mode 100644 index 0000000..bfe80fe --- /dev/null +++ b/doc/job/category.md @@ -0,0 +1,21 @@ +# category + +Meaning: +- Every test has its category, the category field is a test type of test cases. +- When you write a job yaml, you can chose one of benchmark, functional, noise-benchmark + to assign the value of category. +- Here are the difference of difference category: + 1. If category is benchmark, it will monitor some system information: + kmsg: print the kernel startup information + boot-time: start time diagnosis item + uptime: displays how long the system has been running + iostat: print the time when the data is collected + heartbeat: provide the heartbeat monitors service + ... + other monitors you can see them in $LKP_SRC/include/category/benchmark + 2. If category is functional, it will monitor some system information: + kmsg: print the kernel startup information + heartbeat: provide the heartbeat monitors service + meminfo: solid memory usage status or vm usage status + ... + you can see the monitors in $LKP_SRC/include/category/functional -- 2.23.0

1 0

[PATCH v3 lkp-tests] etup/sysctl: add system parameters
by Zhang Yu 26 Nov '20

26 Nov '20

[why] When testing need to setup system parameters. [how] Add parameters to setup/sysctl. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- setup/sysctl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/sysctl b/setup/sysctl index 81374939..2ace4db2 100755 --- a/setup/sysctl +++ b/setup/sysctl @@ -202,6 +202,9 @@ # - sc_netdev_max_backlog # - sc_rmem_max # - sc_wmem_max +# - sc_ipv4_tcp_rmem +# - sc_ipv4_tcp_wmem +# - sc_sched_features . $LKP_SRC/lib/reproduce-log.sh @@ -405,8 +408,11 @@ [ -n "$sc_ipv4_tcp_max_tw_buckets" ] && log_eval "echo '$sc_ipv4_tcp_max_tw_buckets' > '/proc/sys/net/ipv4/tcp_max_tw_buckets'" [ -n "$sc_ipv4_tcp_max_syn_backlog" ] && log_eval "echo '$sc_ipv4_tcp_max_syn_backlog' > '/proc/sys/net/ipv4/tcp_max_syn_backlog'" [ -n "$sc_ipv4_ip_local_port_range" ] && log_eval "echo '$sc_ipv4_ip_local_port_range' > '/proc/sys/net/ipv4/ip_local_port_range'" +[ -n "$sc_ipv4_tcp_rmem" ] && log_eval "echo '$sc_ipv4_tcp_rmem' > '/proc/sys/net/ipv4/tcp_rmem'" +[ -n "$sc_ipv4_tcp_wmem" ] && log_eval "echo '$sc_ipv4_tcp_wmem' > '/proc/sys/net/ipv4/tcp_wmem'" [ -n "$sc_netdev_max_backlog" ] && log_eval "echo '$sc_netdev_max_backlog' > '/proc/sys/net/core/netdev_max_backlog'" [ -n "$sc_rmem_max" ] && log_eval "echo '$sc_rmem_max' > '/proc/sys/net/core/rmem_max'" [ -n "$sc_wmem_max" ] && log_eval "echo '$sc_wmem_max' > '/proc/sys/net/core/wmem_max'" +[ -n "$sc_sched_features" ] && log_eval "echo '$sc_sched_features' > '/sys/kernel/debug/sched_features'" exit 0 -- 2.23.0

1 0

[PATCH v2 compass-ci] container/manjaro: fix failed to build image in x86 machine
by Liu Yinsi 26 Nov '20

26 Nov '20

[why] when execute manjaro/build in x86_machine, error message: standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error"The command '/bin/sh -c chmod 755 /etc /etc/pacman.d' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/manjaro/Dockerfile | 7 +++++-- container/manjaro/build | 11 ++++++++++- .../root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 .../root/{ => aarch64}/etc/pacman.d/mirrors/China | 0 container/manjaro/root/x86_64/etc/pacman.d/mirrorlist | 1 + .../manjaro/root/x86_64/etc/pacman.d/mirrors/China | 1 + 6 files changed, 17 insertions(+), 3 deletions(-) rename container/manjaro/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) rename container/manjaro/root/{ => aarch64}/etc/pacman.d/mirrors/China (100%) create mode 100644 container/manjaro/root/x86_64/etc/pacman.d/mirrorlist create mode 100644 container/manjaro/root/x86_64/etc/pacman.d/mirrors/China diff --git a/container/manjaro/Dockerfile b/container/manjaro/Dockerfile index 48c5ee6..762acc5 100644 --- a/container/manjaro/Dockerfile +++ b/container/manjaro/Dockerfile @@ -1,11 +1,14 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM manjaroarm/manjaro-aarch64-base +ARG BASE_IMAGE + +FROM ${BASE_IMAGE} MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> +ARG ARCH -COPY root / +COPY root/${ARCH} / RUN chmod 755 /etc /etc/pacman.d RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim diff --git a/container/manjaro/build b/container/manjaro/build index 3ce3049..bc3760d 100755 --- a/container/manjaro/build +++ b/container/manjaro/build @@ -2,4 +2,13 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t manjaro:testbed . +declare -A BASE_IMAGE_DICT + +BASE_IMAGE_DICT=( + ["aarch64"]="manjaroarm/manjaro-aarch64-base" + ["x86_64"]="manjarolinux/base" +) + +BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} + +docker build -t manjaro:testbed --build-arg ARCH="$(arch)" --build-arg BASE_IMAGE="${BASE_IMAGE}" . diff --git a/container/manjaro/root/etc/pacman.d/mirrorlist b/container/manjaro/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/manjaro/root/etc/pacman.d/mirrorlist rename to container/manjaro/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/manjaro/root/etc/pacman.d/mirrors/China b/container/manjaro/root/aarch64/etc/pacman.d/mirrors/China similarity index 100% rename from container/manjaro/root/etc/pacman.d/mirrors/China rename to container/manjaro/root/aarch64/etc/pacman.d/mirrors/China diff --git a/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist b/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..a9def4b --- /dev/null +++ b/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch/ diff --git a/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China b/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China new file mode 100644 index 0000000..a9def4b --- /dev/null +++ b/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China @@ -0,0 +1 @@ +Server = http://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch/ -- 2.23.0

1 0

[PATCH compass-ci 3/5] openeuler: the function executed in docker
by Wang Chenglong 26 Nov '20

26 Nov '20

Install packages, set rootfs, pack images package and other actions in docker. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 40 ++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 container/osimage/openeuler/create-image diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image new file mode 100755 index 0000000..58fe0b1 --- /dev/null +++ b/container/osimage/openeuler/create-image @@ -0,0 +1,40 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +setup_rootfs() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + rm -f /etc/machine-id + rm -f /var/lib/dbus/machine-id + systemd-machine-id-setup +} + + +setup_login() { + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + chmod 644 /tmp/${IMAGE_NAME} +} + +install_packages +setup_rootfs +setup_login +pack_cgz -- 2.23.0

2 2

[PATCH lkp-tests] stats/mysql-server: fix rubocop offenses
by Zhang Yu 26 Nov '20

26 Nov '20

Inspecting 1 file Offenses: mysql-server:18:17: C: Style/RedundantRegexpEscape: Redundant escape inside regexp literal when /\[\s*\d+\%\]/ ^^ Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/mysql-server | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/mysql-server b/stats/mysql-server index d3f3fd79..037d1f0f 100755 --- a/stats/mysql-server +++ b/stats/mysql-server @@ -15,7 +15,7 @@ while (line = STDIN.gets) # 200706 14:35:48 [ 99%] main.subquery_sj_innodb_all w8 [ pass ] 140 # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 - when /\[\s*\d+\%\]/ + when /\[\s*\d+%\]/ results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') -- 2.23.0

1 0

[PATCH compass-ci 5/5] openeuler: the link for one command to build openeuler initramfs image
by Wang Chenglong 26 Nov '20

26 Nov '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- .../openeuler/aarch64/20.03-LTS/build | 30 +++++++++++++++++++ .../initramfs/openeuler/aarch64/20.09/build | 1 + 2 files changed, 31 insertions(+) create mode 100755 rootfs/initramfs/openeuler/aarch64/20.03-LTS/build create mode 120000 rootfs/initramfs/openeuler/aarch64/20.09/build diff --git a/rootfs/initramfs/openeuler/aarch64/20.03-LTS/build b/rootfs/initramfs/openeuler/aarch64/20.03-LTS/build new file mode 100755 index 0000000..016f8f9 --- /dev/null +++ b/rootfs/initramfs/openeuler/aarch64/20.03-LTS/build @@ -0,0 +1,30 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +#. ${CCI_SRC}/container/osimage/centos/lib +#. ${CCI_SRC}/container/defconfig.sh +. $HOME/compass-ci/container/osimage/openeuler/lib +. $HOME/compass-ci/container/defconfig.sh + +export_root_passwd +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -w / + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/files-to-exclude:/tmp/files-to-exclude + -v $DIR/${image}.repo:/etc/yum.repos.d/${image}.repo + -e ROOT_PASSWD=$ROOT_PASSWD + -e IMAGE_NAME=$IMAGE_NAME + $image + /root/bin/create-image +) + +pull_docker_image +"${cmd[@]}" +docker_cp_image +docker_rm init_docker &> /dev/null +echo "build finished" diff --git a/rootfs/initramfs/openeuler/aarch64/20.09/build b/rootfs/initramfs/openeuler/aarch64/20.09/build new file mode 120000 index 0000000..f04acac --- /dev/null +++ b/rootfs/initramfs/openeuler/aarch64/20.09/build @@ -0,0 +1 @@ +../../../../../container/osimage/openeuler/run \ No newline at end of file -- 2.23.0

3 4

[PATCH compass-ci] container/manjaro: fix failed to build image in x86 machine
by Liu Yinsi 26 Nov '20

26 Nov '20

[why] when execute manjaro/build in x86_machine, error message: standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error"The command '/bin/sh -c chmod 755 /etc /etc/pacman.d' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/manjaro/Dockerfile | 7 +++++-- container/manjaro/build | 11 ++++++++++- .../root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 .../root/{ => aarch64}/etc/pacman.d/mirrors/China | 0 container/manjaro/root/x86_64/etc/pacman.d/mirrorlist | 1 + .../manjaro/root/x86_64/etc/pacman.d/mirrors/China | 1 + 6 files changed, 17 insertions(+), 3 deletions(-) rename container/manjaro/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) rename container/manjaro/root/{ => aarch64}/etc/pacman.d/mirrors/China (100%) create mode 100644 container/manjaro/root/x86_64/etc/pacman.d/mirrorlist create mode 100644 container/manjaro/root/x86_64/etc/pacman.d/mirrors/China diff --git a/container/manjaro/Dockerfile b/container/manjaro/Dockerfile index 48c5ee6..762acc5 100644 --- a/container/manjaro/Dockerfile +++ b/container/manjaro/Dockerfile @@ -1,11 +1,14 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM manjaroarm/manjaro-aarch64-base +ARG BASE_IMAGE + +FROM ${BASE_IMAGE} MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> +ARG ARCH -COPY root / +COPY root/${ARCH} / RUN chmod 755 /etc /etc/pacman.d RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim diff --git a/container/manjaro/build b/container/manjaro/build index 3ce3049..114e550 100755 --- a/container/manjaro/build +++ b/container/manjaro/build @@ -2,4 +2,13 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t manjaro:testbed . +declare -A BASE_IMAGE_DICT + +BASE_IMAGE_DICT=( + ["aarch64"]="manjaroarm/manjaro-aarch64-base" + ["x86_64"]="manjarolinux/base" +) + +BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} + +docker build -t manjaro:testbed --build-arg ARCH=$(arch) --build-arg BASE_IMAGE="${BASE_IMAGE}" . diff --git a/container/manjaro/root/etc/pacman.d/mirrorlist b/container/manjaro/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/manjaro/root/etc/pacman.d/mirrorlist rename to container/manjaro/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/manjaro/root/etc/pacman.d/mirrors/China b/container/manjaro/root/aarch64/etc/pacman.d/mirrors/China similarity index 100% rename from container/manjaro/root/etc/pacman.d/mirrors/China rename to container/manjaro/root/aarch64/etc/pacman.d/mirrors/China diff --git a/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist b/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..a9def4b --- /dev/null +++ b/container/manjaro/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch/ diff --git a/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China b/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China new file mode 100644 index 0000000..a9def4b --- /dev/null +++ b/container/manjaro/root/x86_64/etc/pacman.d/mirrors/China @@ -0,0 +1 @@ +Server = http://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch/ -- 2.23.0

1 0

[PATCH v3 lkp-tests] spec/submit: add job yaml for multi-args test
by Bai Jing 26 Nov '20

26 Nov '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 9 +++++++ ...unixbench-spec-1-300-whetstone-double.yaml | 25 +++++++++++++++++++ .../unixbench-spec-3-300-shell1.yaml | 25 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 4 files changed, 63 insertions(+) create mode 100644 spec/submit/matrix/unixbench-spec.yaml create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml new file mode 100644 index 00000000..e17322b4 --- /dev/null +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -0,0 +1,9 @@ +suite: unixbench + +runtime: 300s + +nr_task test: + - 1 whetstone-double + - 3 shell1 + +unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml new file mode 100644 index 00000000..9896bcef --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml new file mode 100644 index 00000000..912d10ee --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..a83d2102 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'link matrix' do + submit_job('matrix') + end end -- 2.23.0

2 1

Re: [PATCH compass-ci 3/5] openeuler: the function executed in docker
by Xiao Shenwei 26 Nov '20

26 Nov '20

how about: generate the cgz file in container Reply-To: In-Reply-To: <20201126012754.3857939-3-18509160991(a)163.com> On Thu, Nov 26, 2020 at 09:27:52AM +0800, Wang Chenglong wrote: >Install packages, set rootfs, pack images package and >other actions in docker. > >Signed-off-by: Wang Chenglong <18509160991(a)163.com> >--- > container/osimage/openeuler/create-image | 40 ++++++++++++++++++++++++ > 1 file changed, 40 insertions(+) > create mode 100755 container/osimage/openeuler/create-image > >diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image >new file mode 100755 >index 0000000..58fe0b1 >--- /dev/null >+++ b/container/osimage/openeuler/create-image >@@ -0,0 +1,40 @@ >+#!/bin/bash >+# SPDX-License-Identifier: MulanPSL-2.0+ >+# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. >+ >+install_packages() >+{ >+ yum repolist >+ yum -y install --skip-broken $(</tmp/packages-to-install) >+ yum clean all >+} >+ where use /tmp/file-to-exclude Thanks, Shenwei >+setup_rootfs() >+{ >+ ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime >+ rm -f /etc/machine-id >+ rm -f /var/lib/dbus/machine-id >+ systemd-machine-id-setup >+} >+ >+ >+setup_login() { >+ [ -n "$ROOT_PASSWD" ] || return >+ echo "Changing root password" >+ passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") >+ sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" >+ >+ sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" >+} >+ >+pack_cgz() >+{ >+ echo "Packing package. Please wait." >+ find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} >+ chmod 644 /tmp/${IMAGE_NAME} >+} >+ >+install_packages >+setup_rootfs >+setup_login >+pack_cgz >-- >2.23.0 >

2 1

[PATCH v3 lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..b9b6c8f1 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,18 @@ +#!/usr/bin/env crystal + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + next unless line =~ /transactions:.*\(\s*(\S+)/ + + puts "transactions: #{$1}" +end -- 2.23.0

2 3

[PATCH v3 lkp-tests 2/3] stats/crystal: mysql-server to mysql-server.cr
by Lu Kaiyi 26 Nov '20

26 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/mysql-server.cr | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 stats/crystal/mysql-server.cr diff --git a/stats/crystal/mysql-server.cr b/stats/crystal/mysql-server.cr new file mode 100644 index 00000000..e2ab4b8e --- /dev/null +++ b/stats/crystal/mysql-server.cr @@ -0,0 +1,21 @@ +#!/usr/bin/env crystal +# frozen_string_literal: true + +results = [] of String + +while (line = STDIN.gets) + case line + # The output is as below: + # 200706 14:35:48 [ 99%] main.subquery_sj_innodb_all w8 [ pass ] 140 + # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 + # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 + when /\[\s*\d+\%\]/ + results << line.split(']')[1].split[0] + ".pass: 1" if line.includes?("[ pass ]") + results << line.split(']')[1].split[0] + ".fail: 1" if line.includes?("[ fail ]") + results << line.split(']')[1].split[0] + ".skip: 1" if line.includes?("[ skipped ]") + end +end + +results.each do |item| + puts item +end -- 2.23.0

2 1

[PATCH v3 lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to separate job.yaml, it contains several yaml documents separated by "---" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 19 +++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 27 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 104 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..fbb9dc55 --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,19 @@ +suite: ftq + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..f3e2c553 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: +monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..63bac5cf --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: +monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..a802dc1b --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,27 @@ +--- +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ce6fed25 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

1 0

[PATCH v4 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to merge job.yaml, it support merging from external file by "<<: job.yaml" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/atomic.yaml | 3 +++ .../merge_yaml/atomic/atomic-defaults.yaml | 19 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 19 +++++++++++++++++++ spec/submit_spec.rb | 4 ++++ 5 files changed, 46 insertions(+) create mode 100644 spec/submit/merge_yaml/atomic.yaml create mode 100644 spec/submit/merge_yaml/atomic/atomic-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/atomic.yaml b/spec/submit/merge_yaml/atomic.yaml new file mode 100644 index 00000000..331cb470 --- /dev/null +++ b/spec/submit/merge_yaml/atomic.yaml @@ -0,0 +1,3 @@ +suite: atomic + +atomic: diff --git a/spec/submit/merge_yaml/atomic/atomic-defaults.yaml b/spec/submit/merge_yaml/atomic/atomic-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/atomic/atomic-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..30019c6c --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: atomic.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/crystal: add libmicro.cr
by Bai Jing 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/crystal/libmicro.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 stats/crystal/libmicro.cr diff --git a/stats/crystal/libmicro.cr b/stats/crystal/libmicro.cr new file mode 100644 index 00000000..2a3f42f0 --- /dev/null +++ b/stats/crystal/libmicro.cr @@ -0,0 +1,10 @@ +#!/usr/bin/env crystal + +# prc thr usecs/call samples errors cnt/samp +# close_tmp 1 1 0.61880 201 0 640 + +while (line = STDIN.gets) + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + + puts "#{$1}: #{$2}" +end -- 2.23.0

1 0

[PATCH lkp-tests] stats/crystal: add libmicro.cr
by Bai Jing 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/crystal/libmicro.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100755 stats/crystal/libmicro.cr diff --git a/stats/crystal/libmicro.cr b/stats/crystal/libmicro.cr new file mode 100755 index 00000000..2a3f42f0 --- /dev/null +++ b/stats/crystal/libmicro.cr @@ -0,0 +1,10 @@ +#!/usr/bin/env crystal + +# prc thr usecs/call samples errors cnt/samp +# close_tmp 1 1 0.61880 201 0 640 + +while (line = STDIN.gets) + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + + puts "#{$1}: #{$2}" +end -- 2.23.0

1 1

[PATCH v3 lkp-tests] lkp-tests/*: update AUR Helper for archlinux
by Wang Yong 26 Nov '20

26 Nov '20

yaourt is unmaintained and yay also support native pacman operation, so only minor changes are needed Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 15 +++++++++++++++ distro/installer/archlinux | 15 +++++++++------ lib/install.sh | 2 +- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/distro/archlinux b/distro/archlinux index 408c5fa23..3f8d62ebc 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -109,3 +109,18 @@ pack_benchmark() ) done } + +add_aur_helper() +{ + echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + pacman -Q yay 2>/dev/null && return + + local user=lkp + local dir=/home/${user} + + mkdir -p ${dir} && chown ${user}:${user} ${dir} + pacman -S --needed --noconfirm git base-devel 2>/dev/null + su - ${user} -c "git clone https://aur.archlinux.org/yay.git" + su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" +} diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 27cdc63cf..3be9e7953 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -2,12 +2,15 @@ # Archlinux package installation -# yaourt is a wrapper for pacman that adds automated access -# to the AUR. -# See: https://wiki.archlinux.org/index.php/Yaourt +# Yet Another Yogurt - An AUR Helper Written in Go +# Yay is based on the design of yaourt, apacman and pacaur +# See: https://github.com/Jguer/yay -# yaourt not support root to build package +# Yay not support root to build package # Use regular user to do this -regular_user=`grep 1000 /etc/passwd | awk -F':' '{ print $1 }'` +. $LKP_SRC/distro/${DISTRO} +add_aur_helper -sudo -u $regular_user yaourt -S --noconfirm $* +regular_user=$(grep 1090 /etc/passwd | awk -F':' '{ print $1 }') + +sudo -u $regular_user yay -S --needed --noconfirm $* diff --git a/lib/install.sh b/lib/install.sh index 881ade9f4..bf7bf21d7 100755 --- a/lib/install.sh +++ b/lib/install.sh @@ -17,7 +17,7 @@ sync_distro_sources() yum update fi ;; redhat) yum update ;; - archlinux) yaourt -Sy ;; + archlinux) yay -Sy ;; opensuse) zypper update ;; oracle) yum update ;; -- 2.23.0

1 0

[PATCH lkp-tests 2/2] spec/submit/link_jobs: update monitors value
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml | 2 +- spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml | 2 +- spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml index 3f26d1ae..9a613c00 100644 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml @@ -26,4 +26,4 @@ pp: freq: 1000 test: add cpufreq_governor: -monitors: {} +monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml index 87945a00..8bf52d0f 100644 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml @@ -26,4 +26,4 @@ pp: freq: 10000 test: add cpufreq_governor: -monitors: {} +monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml index 247cdc37..8cb78287 100644 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml @@ -26,4 +26,4 @@ pp: freq: 20 test: cache cpufreq_governor: -monitors: {} +monitors: -- 2.23.0

1 0

[PATCH lkp-tests 1/2] lib/job.rb: set monitors to nil when no category input
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index c8893433..fe373e86 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -584,8 +584,9 @@ class Job def add_monitors @job[comment_to_symbol('auto generated by add_monitors()')] = nil - @job['monitors'] = {} + @job['monitors'] = nil for_each_in(@job, monitor_params) do |_pk, _h, m_n, _m_args| + @job['monitors'] ||= {} @job['monitors'][m_n] = nil end end -- 2.23.0

1 0

[PATCH compass-ci] doc/job/: add category.md
by Zhang Yu 26 Nov '20

26 Nov '20

Add a document for introducing the category field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/category.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/job/category.md diff --git a/doc/job/category.md b/doc/job/category.md new file mode 100644 index 0000000..9842f58 --- /dev/null +++ b/doc/job/category.md @@ -0,0 +1,21 @@ +# category + +Meaning: +- Every test has its category, the category field is a test type of test cases. +- When you write a job yaml, you can chose one of benchmark, functional, noise-benchmark + to assign the value of category. +- Here are the difference of difference category: + 1. If category: benchmark, default monitors: + kmsg: print the kernel startup information + boot-time: + uptime: start time diafnosis item + iostat: print the time when the data is collected + heartbeat: provide the heartbeat monitors service + ... + other monitors you can see them in $LKP_SRC/include/category/benchmark + 2. If category: functional, default monitors: + kmsg: print the kernel startup information + heartbeat: provide the heartbeat monitors service + meminfo: solid memory usage status or vm usage status + ... + you can see the monitors in $LKP_SRC/include/category/functional -- 2.23.0

2 2

[PATCH lkp-tests] stats/crystal: change the file permissions
by Zhang Yu 26 Nov '20

26 Nov '20

[why] the file permissions should be 664. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mariadb.cr | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 stats/crystal/sysbench-mariadb.cr diff --git a/stats/crystal/sysbench-mariadb.cr b/stats/crystal/sysbench-mariadb.cr old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..1f6709c2 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,17 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + next unless line =~ /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" +end -- 2.23.0

3 3

[PATCH lkp-tests 2/3] iso2rootfs: delete the unnecessary fields
by Yu Chuan 26 Nov '20

26 Nov '20

[Why] 'INITRD_HTTP_HOST' and 'INITRD_HTTP_PORT' should be added by the user on demand at runtime, no need to merge into the git repository. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 -- tests/iso2rootfs | 5 ++--- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 4f3e9c83e6cd..a0ffd280676d 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -7,8 +7,6 @@ iso2rootfs: rootfs_protocol: rootfs_server: rootfs_path: - initrd_http_host: - initrd_http_port: # config rootfs related fields ## supported fields: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 761fbb8b70cf..4a19d42ddd94 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -49,8 +49,6 @@ check_yaml_vars() "rootfs_protocol" "rootfs_server" "rootfs_path" - "initrd_http_host" - "initrd_http_port" ) for yaml_t in "${yaml_vars[@]}" @@ -190,7 +188,8 @@ check_os() get_cgz() { - local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" + + local cgz_url="http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" local cgz_name cgz_name=$(basename "${cgz_url}") local cgz_path_t=${ENV_DIR}/${cgz_name} -- 2.23.0

2 2

[PATCH v2 compass-ci] create define_files in result_root
by Lu Weitao 26 Nov '20

26 Nov '20

[Why] User need debug their result-parse-script which haven't commit in lkp-tests/stats. [How] We already support user write script as "define_file" => {"stats/$script_name" => "$script_content"} in job.yaml, the job.yaml will be save as job to ES DB, write the job['define_file']['stats/$script_name'](value) at $result_root/stats/$script, extract-stats service will parse result by $result_root/stats/$script(user's result-parse-script) firstly Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index bcf6e52..74afee1 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -22,6 +22,8 @@ class StatsWorker if job_id # will consume the job by post-processing job = @es.get_job_content(job_id) result_root = job["result_root"]? + define_files = job["define_files"]? + write_user_stats(result_root.to_s, define_files.as_h) if define_files.is_a?(JSON::Any) result_post_processing(job_id, result_root.to_s, queue_path) @tq.delete_task(queue_path + "/in_process", "#{job_id}") end @@ -139,4 +141,17 @@ class StatsWorker STDERR.puts e.message end end + + def write_user_stats(result_root : String, define_files : Hash(String, JSON::Any)) + return nil unless result_root && File.exists?(result_root) + + define_files.each do |k, v| + if k.match(/^stats/) + Dir.mkdir("#{result_root}/stats") unless File.exists?("#{result_root}/stats") + File.open("#{result_root}/#{k}", "w+", File::Permissions.new(0o775)) do |file| + file.puts v + end + end + end + end end -- 2.23.0

2 2

Re: [PATCH v2 lkp-tests] tats/crystal: add hackbench-git.cr
by Bai Jing 26 Nov '20

26 Nov '20

^^^ stats Thanks, Baijing Reply-To: In-Reply-To: <20201125115826.760733-1-ylzhangah(a)qq.com> On Wed, Nov 25, 2020 at 07:58:26PM +0800, Zhang Yale wrote: >Make ruby2crystal convert > >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > stats/crystal/hackbench-git.cr | 22 ++++++++++++++++++++++ > 1 file changed, 22 insertions(+) > create mode 100644 stats/crystal/hackbench-git.cr >diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr >new file mode 100644 >index 00000000..2c043fc9 >--- /dev/null >+++ b/stats/crystal/hackbench-git.cr >@@ -0,0 +1,22 @@ >+#!/usr/bin/env crystal >+ >+# --------socket thread num=20-------- >+# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 >+# Running with 20*40 (== 800) tasks. >+# Time: 0.495 >+# ... >+ >+time = [] of Float64 >+ >+while (line = gets) >+ case line >+ when /^-+(\w+ \w+ \w+)/ >+ args = $1.tr(" ", "_") >+ when /^Running with .* $== (\d+)$ tasks/ >+ tasks = $1.to_i >+ when /^Time:/ >+ time << line.split[1].to_f >+ end >+end >+ >+puts "#{args}_#{tasks}: #{time.sum / time.size}" >-- >2.23.0 >

2 1

[PATCH v3 lkp-tests 3/3] tests/iozone: refactor format optimization
by Lu Kaiyi 26 Nov '20

26 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/iozone b/tests/iozone index 5186766f..923d8331 100755 --- a/tests/iozone +++ b/tests/iozone @@ -47,8 +47,7 @@ if [ -n "$block_size" ]; then ;; "preadv") args+=" -i 12" ;; - *) echo "invalid test option, please check again!" - exit 1 + *) echo "invalid test option, please check again!" && exit 1 ;; esac done -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/3] stats/mysql-server: remove redundant whitespace
by Lu Kaiyi 26 Nov '20

26 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/mysql-server | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/stats/mysql-server b/stats/mysql-server index d3f3fd79..776f3f2e 100755 --- a/stats/mysql-server +++ b/stats/mysql-server @@ -16,9 +16,9 @@ while (line = STDIN.gets) # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 when /\[\s*\d+\%\]/ - results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') - results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') - results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') + results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') + results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') + results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tats/crystal: add hackbench-git.cr
by Zhang Yale 25 Nov '20

25 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/crystal/hackbench-git.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 stats/crystal/hackbench-git.cr diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr new file mode 100644 index 00000000..2c043fc9 --- /dev/null +++ b/stats/crystal/hackbench-git.cr @@ -0,0 +1,22 @@ +#!/usr/bin/env crystal + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +time = [] of Float64 + +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(" ", "_") + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + time << line.split[1].to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 0

[PATCH lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 25 Nov '20

25 Nov '20

[why] Make ruby2crystal convert. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..1bab78c0 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,19 @@ +#!/usr/bin/env crystal + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + case line + when /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..c604bfa --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,27 @@ +# runtime + +Meaning: +- Seconds to run before exiting. +- Limit the test's run time in $runtime. + +Usage example: + +```bash +#!/bin/sh +# - runtime +``` + +`runtime` is a parameter of test script. +For any test case needing parameter(s), +need such comment line and follow exactly +the same format `# - parameter`. + +```yaml +runtime: 300s +``` + +```yaml +runtime: 1d +sleep: +``` + -- 2.23.0

1 0

[PATCH lkp-tests 3/3] iso2rootfs: add comment to iso2rootfs.yaml
by Yu Chuan 25 Nov '20

25 Nov '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index a0ffd280676d..2c4d3a93ff62 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -1,9 +1,18 @@ suite: iso2rootfs category: benchmark iso2rootfs: + # iso related fields to be used to generate rootfs iso_os: iso_arch: iso_version: + + # place the result rootfs related fields + ## 1. result rootfs will be placed in the following localtion on the + ## remote file server: + ## - {remote_file_server}/{rootfs_path}/{iso_os}/{iso_arch}/ + ## 2. remote file server protocols current supported: + ## - nfs + ## - cifs rootfs_protocol: rootfs_server: rootfs_path: -- 2.23.0

1 0

[PATCH lkp-tests 1/3] iso2rootfs: delete the meaningless field
by Yu Chuan 25 Nov '20

25 Nov '20

[Why] The 'iso2rootfs_cgz' field will always be 'iso2rootfs.cgz', so delete this field, and use the 'iso2rootfs.cgz' directly. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 1 - tests/iso2rootfs | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 1a643b538ac1..4f3e9c83e6cd 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -1,7 +1,6 @@ suite: iso2rootfs category: benchmark iso2rootfs: - iso2rootfs_cgz: iso_os: iso_arch: iso_version: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index fcaadaa1ba46..761fbb8b70cf 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -43,7 +43,6 @@ check_yaml_vars() "os" "os_arch" "os_version" - "iso2rootfs_cgz" "iso_os" "iso_arch" "iso_version" @@ -191,7 +190,7 @@ check_os() get_cgz() { - local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/${iso2rootfs_cgz}" + local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" local cgz_name cgz_name=$(basename "${cgz_url}") local cgz_path_t=${ENV_DIR}/${cgz_name} -- 2.23.0

1 0

[PATCH v5 lkp-tests] job: fix each_jobs to fit multi-args
by Bai Jing 25 Nov '20

25 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h2 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} Because "@job2=hash" will merge the h2 to @job after "each_job". We need to merge the h2 to @job before "each_job". So do the merge advance and set @job2={} Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index c8893433..9272392e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -730,7 +730,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..296fc6e --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,20 @@ +# runtime + +Meaning: +Limit run time to runtime seconds. + +Usage example: + +```#/bin/bash +# - runtime +``` + +`runtime` is a parameter of test script. For any test case +needing paramter(s), need such comment line and follow exactly +the same format `# - parameter`. + +job.yaml: + +``` +runtime: 300s +``` -- 2.23.0

2 2

[PATCH lkp-tests] stats/atomic: fix rubocop offenses
by Zhang Yale 25 Nov '20

25 Nov '20

Inspecting 1 file C Offenses: atomic:6:1: C: [Corrected] Layout/TrailingWhitespace: Trailing whitespace detected. 1 file inspected, 1 offense detected, 1 offense corrected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/atomic | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/atomic b/stats/atomic index 71fc4fe6..422d575b 100755 --- a/stats/atomic +++ b/stats/atomic @@ -3,7 +3,7 @@ while (line = STDIN.gets) line.chomp! next unless line =~ /atomic:\s+\d+\s+(\d+).+\s+(\d+)$/ - + puts "threads: #{$1}" puts "score: #{$2}" end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job/: add os-os_verison-os_arch.md
by Zhang Yu 25 Nov '20

25 Nov '20

Add a document for introducing the os, os_version, os_arch fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os-os_verison-os_arch.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/os-os_verison-os_arch.md diff --git a/doc/job/os-os_verison-os_arch.md b/doc/job/os-os_verison-os_arch.md new file mode 100644 index 0000000..9939414 --- /dev/null +++ b/doc/job/os-os_verison-os_arch.md @@ -0,0 +1,16 @@ +# os, os_version, os_arch + +Meaning: +- Each test tells which os it needs, and the os_version, os_arch are also need to be specified, + then the machine will run an os according to these three fields. +- The os, os_version, os_arch are the key for users to specify the os related parameters. +- If os, os_version, os_arch are not given by users, it will use the default openeuler os related parameters. +- Here are some examples: + os os_version os_arch + openeuler 20.03, 1.0 aarch64, x86_64 + debian sid, 10 aarch64, x86_64 + centos 7.6, 7.8 aarch64, x86_64 + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g os=openeuler os_arch=aarch64 os_version=20.03 -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..844cdbb --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,20 @@ +# runtime + +Meaning: +Limit run time to runtime seconds. + +Usage example: + +```#/bin/bash +# - runtime +``` + +It explicitly tells that `runtime` are the parameters of test script. +For any test case needing paramter(s), need such comment lines and +follow exactly the same format `# - parameter`. + +job.yaml: + +``` +runtime: 300s +``` -- 2.23.0

1 1

[PATCH compass-ci 4/4] assign-account: build email message for updating configuration
by Luan Shengde 25 Nov '20

25 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../assign-account/build-update-email.rb | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/assign-account/build-update-email.rb diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb new file mode 100755 index 0000000..40d6b3b --- /dev/null +++ b/container/assign-account/build-update-email.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_update_message(email, account_info, conf_info) + message = if conf_info['enable_login'] + + <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] account update notice + + Dear user: + + Your account has been update successfully. + You can use the following command to login the server. + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + + regards + compass-ci + EMAIL_MESSAGE + else + <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] account expiration notice + + Dear user: + + your account has expired. + consult the manager for help if you want to continue using the account. + + regards + compass-ci + EMAIL_MESSAGE + end + + return message +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] assign-account: reattach new dir for the container
by Luan Shengde 25 Nov '20

25 Nov '20

[why] enable execute command 'usermod -s shell' to enable/disable account to login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/start | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/container/assign-account/start b/container/assign-account/start index 2290982..793191a 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -. $CCI_SRC/container/defconfig.sh +. ../defconfig.sh docker_rm assign-account @@ -11,8 +11,7 @@ cmd=( --restart=always --name=assign-account -d - -v /etc/passwd:/etc/passwd:ro - -v /etc/group:/etc/group:ro + -v /etc:/etc:rw -v /srv/account-info:/opt/account_data/:rw -v /home:/home:rw -p 29999:29999 @@ -20,14 +19,3 @@ cmd=( ) "${cmd[@]}" - -cat << EOF -port: 29999 ---- -Client: -./assign-tool.rb email_file - -email subject: "apply ssh account" - -Refer to compass-ci/doc/apply-ssh-account.md for more information. -EOF -- 2.23.0

1 0

[PATCH compass-ci 2/4] assign-account: add new function and optimize
by Luan Shengde 25 Nov '20

25 Nov '20

[why] meet new options for answerback-email meet update configuration function for answerback-email [how] add new function for the new options for answerback-email optimize current function to enable update Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 117 +++++++++++++++---- 1 file changed, 92 insertions(+), 25 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 0e91057..95bec14 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -29,11 +29,14 @@ API: call graph: setup_jumper_account_info - read_account_info - build_account_name + read account + read_account_info + build_account_info + reread_account_info read_jumper_info config_default_yaml config_authorized_key + permit_login_config generate_ssh_key the returned data for setup_jumper_account_info like: @@ -41,12 +44,14 @@ the returned data for setup_jumper_account_info like: "my_login_name" => "login_name", "password" => "password", "jumper_host" => "0.0.0.0", - "jumper_port" => "10000" + "jumper_port" => "10000", + "my_jumper_pubkey" => my_jumper_pubkey } =end require 'fileutils' +require 'yaml' # get jumper and account info class AccountStorage @@ -95,14 +100,21 @@ class AccountStorage end def setup_jumper_account_info - login_name, password = read_account_info + if @data.key?('is_update_account') && @data['is_update_account'] + login_name = @data['my_login_name'] + password = reread_account_info + else + login_name, password = read_account_info + end + jumper_host, jumper_port = read_jumper_info - pub_key = @data['my_ssh_pubkey'][0] - ssh_dir = File.join('/home/', login_name, '.ssh') - config_authorized_key(login_name, pub_key, ssh_dir) unless pub_key.nil? + config_authorized_key(login_name) config_default_yaml(login_name) - my_jumper_pubkey = generate_ssh_key(login_name, ssh_dir) if @data['gen_sshkey'].eql? true + permit_login_config(login_name) + if @data.key?('gen_sshkey') && @data['gen_sshkey'] + my_jumper_pubkey = generate_ssh_key(login_name) + end jumper_account_info = { 'my_login_name' => login_name, @@ -115,34 +127,89 @@ class AccountStorage return jumper_account_info end - def generate_ssh_key(login_name, ssh_dir) + def permit_login_config(login_name) + if @data.key?('enable_login') && @data['enable_login'] + %x(usermod -s /usr/bin/zsh #{login_name}) + else + %x(usermod -s /sbin/nologin #{login_name}) + end + end + + def generate_ssh_key(login_name) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + pub_key_file = File.join(ssh_dir, 'id_rsa.pub') + + return if File.exist?(pub_key_file) && \ + @data['my_ssh_pubkey'].include?(File.read(pub_key_file).strip) + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@account-vm") - %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) - File.read("/home/#{login_name}/.ssh/id_rsa.pub") + + FileUtils.chown_R(login_name, login_name, ssh_dir) + File.read("/home/#{login_name}/.ssh/id_rsa.pub").strip end - def config_default_yaml(login_name) + def touch_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') - FileUtils.mkdir_p default_yaml_dir + FileUtils.mkdir_p default_yaml_dir unless File.exist? default_yaml_dir + + default_yaml = File.join(default_yaml_dir, 'account.yaml') + FileUtils.touch default_yaml unless File.exist? default_yaml + default_yaml + end + def config_default_yaml(login_name) + default_yaml = touch_default_yaml(login_name) + + account_yaml = YAML.load_file(default_yaml) || {} # my_email, my_name, my_uuid is required to config default yaml file # they are added along with 'my_ssh_pubkey' when sending assign account request - File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| - file.puts "my_email: #{@data['my_email']}" - file.puts "my_name: #{@data['my_name']}" - file.puts "my_uuid: #{@data['my_uuid']}" - end - %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + account_yaml['my_email'] = @data['my_email'] + account_yaml['my_name'] = @data['my_name'] + account_yaml['my_uuid'] = @data['my_uuid'] + + f = File.new(default_yaml, 'w') + f.puts account_yaml.to_yaml + f.close + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") + end + + def config_authorized_key(login_name) + pub_key = @data['my_ssh_pubkey'][0] + + return if pub_key.nil? + return if pub_key.strip.end_with?('account-vm') + + ssh_dir = File.join('/home/', login_name, '.ssh') + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + + authorized_file = File.join(ssh_dir, 'authorized_keys') + FileUtils.touch authorized_file unless File.exist? authorized_file + + store_pubkey(ssh_dir, login_name, authorized_file, pub_key) end - def config_authorized_key(login_name, pub_key, ssh_dir) - Dir.mkdir ssh_dir, 0o700 - Dir.chdir ssh_dir - f = File.new('authorized_keys', 'w') + def store_pubkey(ssh_dir, login_name, authorized_file, pub_key) + authorized_keys = File.read(authorized_file).split("\n") + + return if authorized_keys.include?(pub_key.strip) + + f = File.new(authorized_file, 'a') f.puts pub_key f.close - File.chmod 0o600, 'authorized_keys' - %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + + FileUtils.chown_R(login_name, login_name, ssh_dir) + File.chmod 0o600, authorized_file + end + + def reread_account_info + my_login_name_file = File.join(@account_dir, 'assigned-users', @data['my_login_name']) + + message = "No such assigned account exists: #{my_login_name_file}." + raise message unless File.exist? my_login_name_file + + password = File.readlines(my_login_name_file)[0].chomp + + return password end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will user 'rspec spec/submit_spec.rb' to separate job.yaml, it contains several yaml documents separated by "---" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 20 +++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 28 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 108 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..e54adc8b --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,20 @@ +suite: ftq +category: noise-benchmark + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..855e7210 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..78725424 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..40edfcb2 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ce6fed25 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

2 1

[PATCH lkp-tests 1/2] daemon/multi-qemu: fix default parameter format
by Xiao Shenwei 25 Nov '20

25 Nov '20

before: : ${queues:="$tbox_group.$HOSTNAME, $tbox_group.$(arch)"} now: : ${queues:="$tbox_group.$HOSTNAME,$tbox_group.$(arch)"} this parameter will be used by ruby script for ruby command line parameter, if use space like: multi-qemu -q vm-2p8g.taishan200-2280-2s48p-256g--a1, vm-2p8g.aarch64 the vm-2p8g.aarch64 doesn't work correct usage: multi-qemu -q vm-2p8g.taishan200-2280-2s48p-256g--a1,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/multi-qemu | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 7cc625f4..aa2348b4 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -6,7 +6,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${queues:="$tbox_group.$HOSTNAME, $tbox_group.$(arch)"} +: ${queues:="$tbox_group.$HOSTNAME,$tbox_group.$(arch)"} + multi_qemu() { export CCI_SRC=/c/compass-ci -- 2.23.0

2 1

[PATCH compass-ci] job.cr: delete 2 variables from METHOD_KEYS
by Xu Xijian 25 Nov '20

25 Nov '20

Since linux_modules_initrd and linux_headers_initrd are only used in initramfs, they don't need to be initialized in all jobs, so delete them from METHOD_KEYS. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a9923cc..378fd00 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -97,8 +97,6 @@ class Job docker_image kernel_version linux_vmlinuz_path - linux_modules_initrd - linux_headers_initrd ) macro method_missing(call) @@ -390,9 +388,9 @@ class Job temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" temp_initrds << "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{linux_modules_initrd}")}" + "#{JobHelper.service_path(self["linux_modules_initrd"])}" temp_initrds << "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{linux_headers_initrd}")}" + "#{JobHelper.service_path(self["linux_headers_initrd"])}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

1 0

[PATCH lkp-tests 2/2] lib/bootstrap.sh: fix redirect_stdout_stderr for docker
by Xiao Shenwei 25 Nov '20

25 Nov '20

refer-to: compass-ci: c7c4ba94b3ef00a3f5c91200f85ea2d924726132 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/bootstrap.sh | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ce34a632..46e77e72 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -270,11 +270,7 @@ announce_bootup() redirect_stdout_stderr() { - if is_docker; then - exec > /tmp/stdout - exec 2> /tmp/stderr - return - fi + is_docker && return [ -c /dev/kmsg ] || return has_cmd tail || return -- 2.23.0

1 0

[PATCH compass-ci] provider/docker: forward correct and error logs to docker log collector
by Xiao Shenwei 25 Nov '20

25 Nov '20

[why] $LKP_SRC/lib/bootstrap.sh:271 redirect_stdout_stderr() result file: output stderr stdout these file will based on /tmp/stdout and /tmp/stderr the function will execute: exec > /tmp/stdout exec 2> /tmp/stderr docker log based on /dev/stdout and /dev/stderr, so docker log can't collect the job's output. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..3d04406 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap +# forward correct and error logs to docker log collector +touch /tmp/stderr /tmp/strout +tail -f /tmp/stdout > /dev/stdout & +echo $! >> /tmp/pid-tail-global +tail -f /tmp/stderr > /dev/stderr & +echo $! >> /tmp/pid-tail-global + +sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap 1> /tmp/stdout 2> /tmp/stderr -- 2.23.0

1 0

[PATCH v4 lkp-tests] daemon/sshd: change pub_key to my_ssh_pubkey
by Zhang Yale 25 Nov '20

25 Nov '20

[why] 1. "my_ssh_pubkey" field is a collection of all pub_keys submitted by users. 2. We can save all the pub_keys submitted by users in the ES database and password-free login. 3. Therefore, users can sometimes borrow a machine to test without providing a pub_key. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index a3fdf954..c7d7b60c 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,5 +1,5 @@ #!/bin/sh -# pub_key +# my_ssh_pubkey # my_email # sshr_port # sshr_port_len @@ -14,10 +14,10 @@ sshr_ip=$LKP_SERVER run_ssh() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh - echo "$pub_key" > /root/.ssh/authorized_keys + echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys systemctl start sshd } @@ -73,11 +73,11 @@ compose_email() set_port() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return for i in $(seq 1 10) do - port=$(($(date +%s%N)%${sshr_port_len}+${sshr_port_base})) + port=$(($(date +%s%N)%$sshr_port_len+$sshr_port_base)) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return -- 2.23.0

1 0

[PATCH compass-ci] block_helper.cr: refactor if block into one line
by Xu Xijian 25 Nov '20

25 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/block_helper.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/lib/block_helper.cr b/src/lib/block_helper.cr index 37ad367..596ef7e 100644 --- a/src/lib/block_helper.cr +++ b/src/lib/block_helper.cr @@ -28,9 +28,7 @@ class BlockHelper @block_helper[uuid] = fiber end - if yield == true - spawn fiber.run - end + spawn fiber.run if yield == true until fiber.dead? Fiber.yield -- 2.23.0

1 0

[PATCH lkp-tests] submit: fix the processes of option: --my-queue
by Ren Wen 25 Nov '20

25 Nov '20

If use options '--my-queue' and '-o' at the same time, the output job file will not contain the value of the covered field 'queue'. So pass the 'job' not job_hash. Signed-off-by: Ren Wen <15991987063(a)163.com> --- sbin/submit | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index be588bed..74815202 100755 --- a/sbin/submit +++ b/sbin/submit @@ -139,12 +139,13 @@ ARGV.each do |jobfile| sh_hash['job2sh']['on_fail'] = sh_on_fail end + add_my_queue(job) if opt_my_queue + # merge job info job_hash = job.to_hash job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) - add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

2 1

[PATCH v2 lkp-tests] tests/build-pkg: update mount option
by Lin Jiaxin 25 Nov '20

25 Nov '20

[why] refer-to: compass-ci: commit 96584b45e528a4bad1f9 [how] add port=446 option to mount.cifs Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 65c83e22..29d35997 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -37,7 +37,7 @@ mount_dest() mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } -- 2.23.0

1 0

[PATCH lkp-tests] lkp-tests/build-pkg: update mount option
by Lin Jiaxin 25 Nov '20

25 Nov '20

[why] refer-to: compass-ci: commit 96584b45e528a4bad1f9 [how] add port=446 option to mount.cifs Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 65c83e22..29d35997 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -37,7 +37,7 @@ mount_dest() mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to separate job.yaml, it contains several yaml documents separated by "---" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 20 +++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 28 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 108 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..e54adc8b --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,20 @@ +suite: ftq +category: noise-benchmark + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..855e7210 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..78725424 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..40edfcb2 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ce6fed25 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will use 'rspec spec/submit_spec.rb' to merge job.yaml, it support merging from external file by "<<: job.yaml" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/atomic.yaml | 4 ++ .../merge_yaml/atomic/atomic-defaults.yaml | 72 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 71 ++++++++++++++++++ spec/submit_spec.rb | 4 ++ 5 files changed, 152 insertions(+) create mode 100644 spec/submit/merge_yaml/atomic.yaml create mode 100644 spec/submit/merge_yaml/atomic/atomic-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/atomic.yaml b/spec/submit/merge_yaml/atomic.yaml new file mode 100644 index 00000000..d0927923 --- /dev/null +++ b/spec/submit/merge_yaml/atomic.yaml @@ -0,0 +1,4 @@ +suite: atomic +category: benchmark + +atomic: diff --git a/spec/submit/merge_yaml/atomic/atomic-defaults.yaml b/spec/submit/merge_yaml/atomic/atomic-defaults.yaml new file mode 100644 index 00000000..920ebec2 --- /dev/null +++ b/spec/submit/merge_yaml/atomic/atomic-defaults.yaml @@ -0,0 +1,72 @@ +--- +suite: atomic +category: benchmark +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: {} + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..30019c6c --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: atomic.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..fc9e073a --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler-taskqueue: use before_all
by Wu Zhende 25 Nov '20

25 Nov '20

use before_all to replace the same content in each API Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/scheduler.cr | 18 ++++-------------- src/taskqueue/taskqueue.cr | 10 ++++------ 2 files changed, 8 insertions(+), 20 deletions(-) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index a946ea0..5f97567 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -32,6 +32,10 @@ module Scheduler sched = Sched.new + before_all do |env| + env.response.headers["Connection"] = "close" + end + # for debug (maybe kemal debug|logger does better) def self.debug_message(env, response) puts %({"from": "#{env.request.remote_address}", "response": #{response.to_json}}) @@ -39,7 +43,6 @@ module Scheduler # echo alive get "/" do |env| - env.response.headers["Connection"] = "close" debug_message(env, "Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{VERSION}" end @@ -50,7 +53,6 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - env.response.headers["Connection"] = "close" response = sched.find_job_boot(env) job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? @@ -61,7 +63,6 @@ module Scheduler # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - env.response.headers["Connection"] = "close" response = sched.find_next_job_boot(env) job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? @@ -74,7 +75,6 @@ module Scheduler # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - env.response.headers["Connection"] = "close" job_messages = sched.submit_job(env) job_messages.each do |job_message| @@ -86,7 +86,6 @@ module Scheduler # file download server get "/job_initrd_tmpfs/:job_id/:job_package" do |env| - env.response.headers["Connection"] = "close" job_id = env.params.url["job_id"] job_package = env.params.url["job_package"] file_path = ::File.join [Kemal.config.public_folder, job_id, job_package] @@ -103,7 +102,6 @@ module Scheduler # # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| - env.response.headers["Connection"] = "close" if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) sched.set_host_mac(client_mac, client_hostname) @@ -115,7 +113,6 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| - env.response.headers["Connection"] = "close" if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) sched.set_host2queues(client_host, client_queues) @@ -127,7 +124,6 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| - env.response.headers["Connection"] = "close" if client_mac = env.params.query["mac"]? sched.del_host_mac(client_mac) @@ -139,7 +135,6 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| - env.response.headers["Connection"] = "close" if client_host = env.params.query["host"]? sched.del_host2queues(client_host) @@ -155,7 +150,6 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - env.response.headers["Connection"] = "close" puts sched.update_job_parameter(env) "Done" @@ -179,7 +173,6 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - env.response.headers["Connection"] = "close" response = sched.request_cluster_state(env) debug_message(env, response) @@ -191,7 +184,6 @@ module Scheduler # /~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job.yaml&job_id=40 # curl "http://localhost:3000/~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job…" get "/~lkp/cgi-bin/lkp-post-run" do |env| - env.response.headers["Connection"] = "close" # get job_id from request job_id = env.params.query["job_id"]? if job_id @@ -202,14 +194,12 @@ module Scheduler end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - env.response.headers["Connection"] = "close" puts sched.update_tbox_wtmp(env) "Done" end get "/~lkp/cgi-bin/report_ssh_port" do |env| - env.response.headers["Connection"] = "close" testbox = env.params.query["tbox_name"] ssh_port = env.params.query["ssh_port"].to_s job_id = env.params.query["job_id"].to_s diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index f41cfa8..0c82333 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -10,6 +10,10 @@ require "./queue" class TaskQueue VERSION = "0.0.2" + before_all do |env| + env.response.headers["Connection"] = "close" + end + # l2pH: limit 2 / hours # l100pms: limit 100 / ms @@rate_limiter = RateLimiter(String).new @@ -42,7 +46,6 @@ class TaskQueue # # response: TaskQueue(a)v0.0.1 is alive. get "/" do |env| - env.response.headers["Connection"] = "close" response = "TaskQueue@v#{VERSION} is alive." debug_message(env, response, Time.utc) @@ -61,7 +64,6 @@ class TaskQueue # 400 "Missing parameter <queue>" # 400 "Missing http body" post "/add" do |env| - env.response.headers["Connection"] = "close" response = queue_respond_add(env) debug_message(env, response, Time.utc) response if env.response.status_code == 200 @@ -75,7 +77,6 @@ class TaskQueue # 201 ## when there has no task in queue (scheduler/$tbox_group) # 400 "Missing parameter <queue>" put "/consume" do |env| - env.response.headers["Connection"] = "close" response = queue_respond_consume(env) debug_message(env, response, Time.utc) response if env.response.status_code == 200 @@ -89,7 +90,6 @@ class TaskQueue # 400 "Missing parameter <from|to|id>" # 409 "Can not find id <$id> in queue <scheduler/$tbox_group>" put "/hand_over" do |env| - env.response.headers["Connection"] = "close" response = queue_respond_hand_over(env) debug_message(env, response, Time.utc) nil @@ -103,7 +103,6 @@ class TaskQueue # 400 "Missing parameter <queue|id>" # 409 "Can not find id <$id> in queue <scheduler/$tbox_group>" put "/delete" do |env| - env.response.headers["Connection"] = "close" response = queue_respond_delete(env) debug_message(env, response, Time.utc) nil @@ -119,7 +118,6 @@ class TaskQueue # 400 "Missing parameter <queue>" # 413 "Query results too large keys" get "/keys" do |env| - env.response.headers["Connection"] = "close" response = queue_respond_keys(env) # debug_message(env, response, Time.utc) # maybe too large response.to_json unless env.response.status_code == 201 -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/2] tests/iozone: add invalid test option identification
by Lu Kaiyi 25 Nov '20

25 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/iozone b/tests/iozone index bafc0087..5186766f 100755 --- a/tests/iozone +++ b/tests/iozone @@ -47,6 +47,9 @@ if [ -n "$block_size" ]; then ;; "preadv") args+=" -i 12" ;; + *) echo "invalid test option, please check again!" + exit 1 + ;; esac done else -- 2.23.0

4 3

[PATCH lkp-tests] tests/iozone: refactor format optimization
by Lu Kaiyi 25 Nov '20

25 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/iozone b/tests/iozone index 5186766f..923d8331 100755 --- a/tests/iozone +++ b/tests/iozone @@ -47,8 +47,7 @@ if [ -n "$block_size" ]; then ;; "preadv") args+=" -i 12" ;; - *) echo "invalid test option, please check again!" - exit 1 + *) echo "invalid test option, please check again!" && exit 1 ;; esac done -- 2.23.0

2 1

[PATCH v2 lkp-tests] setup/sysctl: add system parameters
by Zhang Yu 25 Nov '20

25 Nov '20

[why] When testing need to setup system parameters. [how] Add parameters to setup/sysctl. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- setup/sysctl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/sysctl b/setup/sysctl index 81374939..1085df49 100755 --- a/setup/sysctl +++ b/setup/sysctl @@ -202,6 +202,9 @@ # - sc_netdev_max_backlog # - sc_rmem_max # - sc_wmem_max +# - sc_tcp_rmem +# - sc_tcp_wmem +# - sc_sched_features . $LKP_SRC/lib/reproduce-log.sh @@ -405,8 +408,11 @@ [ -n "$sc_ipv4_tcp_max_tw_buckets" ] && log_eval "echo '$sc_ipv4_tcp_max_tw_buckets' > '/proc/sys/net/ipv4/tcp_max_tw_buckets'" [ -n "$sc_ipv4_tcp_max_syn_backlog" ] && log_eval "echo '$sc_ipv4_tcp_max_syn_backlog' > '/proc/sys/net/ipv4/tcp_max_syn_backlog'" [ -n "$sc_ipv4_ip_local_port_range" ] && log_eval "echo '$sc_ipv4_ip_local_port_range' > '/proc/sys/net/ipv4/ip_local_port_range'" +[ -n "$sc_tcp_rmem" ] && log_eval "echo '$sc_tcp_rmem' > '/proc/sys/net/ipv4/tcp_rmem'" +[ -n "$sc_tcp_wmem" ] && log_eval "echo '$sc_tcp_wmem' > '/proc/sys/net/ipv4/tcp_wmem'" [ -n "$sc_netdev_max_backlog" ] && log_eval "echo '$sc_netdev_max_backlog' > '/proc/sys/net/core/netdev_max_backlog'" [ -n "$sc_rmem_max" ] && log_eval "echo '$sc_rmem_max' > '/proc/sys/net/core/rmem_max'" [ -n "$sc_wmem_max" ] && log_eval "echo '$sc_wmem_max' > '/proc/sys/net/core/wmem_max'" +[ -n "$sc_sched_features" ] && log_eval "echo '$sc_sched_features' > '/sys/kernel/debug/sched_features'" exit 0 -- 2.23.0

1 0

[PATCH lkp-tests] daemon/sshd: fix pub_key to my_ssh_pubkey
by Zhang Yale 25 Nov '20

25 Nov '20

[why] - "my_ssh_pubkey" field is a set of all pub_keys that had been submitted by users. - We can save all the pub_keys submitted by users in the ES database and set the login mode to password-free. - Therefore, users can sometimes borrow a machine to test without providing a pub_key. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index a3fdf954..7919f81d 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,5 +1,5 @@ #!/bin/sh -# pub_key +# my_ssh_pubkey # my_email # sshr_port # sshr_port_len @@ -14,10 +14,10 @@ sshr_ip=$LKP_SERVER run_ssh() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh - echo "$pub_key" > /root/.ssh/authorized_keys + echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys systemctl start sshd } @@ -54,7 +54,7 @@ Subject: [NOTIFY Compass-ci] Applying $testbox failed Dear $my_username: Sorry to inform you that your application failed. - You may need to wait a moment, or check whether your pub_key exists. + You may need to wait a moment, or check whether your my_ssh_pubkey exists. Regards Compass-Ci @@ -73,11 +73,11 @@ compose_email() set_port() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return for i in $(seq 1 10) do - port=$(($(date +%s%N)%${sshr_port_len}+${sshr_port_base})) + port=$(($(date +%s%N)%$sshr_port_len+$sshr_port_base)) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return -- 2.23.0

3 4

[PATCH v3 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will give each type of cases for different input and output directories, and support merging from external file. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/atomic.yaml | 4 ++ .../merge_yaml/atomic/atomic-defaults.yaml | 72 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 71 ++++++++++++++++++ spec/submit_spec.rb | 4 ++ 5 files changed, 152 insertions(+) create mode 100644 spec/submit/merge_yaml/atomic.yaml create mode 100644 spec/submit/merge_yaml/atomic/atomic-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/atomic.yaml b/spec/submit/merge_yaml/atomic.yaml new file mode 100644 index 00000000..d0927923 --- /dev/null +++ b/spec/submit/merge_yaml/atomic.yaml @@ -0,0 +1,4 @@ +suite: atomic +category: benchmark + +atomic: diff --git a/spec/submit/merge_yaml/atomic/atomic-defaults.yaml b/spec/submit/merge_yaml/atomic/atomic-defaults.yaml new file mode 100644 index 00000000..920ebec2 --- /dev/null +++ b/spec/submit/merge_yaml/atomic/atomic-defaults.yaml @@ -0,0 +1,72 @@ +--- +suite: atomic +category: benchmark +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: {} + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..30019c6c --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: atomic.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..fc9e073a --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH compass-ci] mail-robot: optimize check url for gitee.com
by Luan Shengde 25 Nov '20

25 Nov '20

[why] use 'git clone repo_url' will clone the whole repo it will need more time and network bandwidth [how] use 'git clone --bare repo_url' instead Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/gitee-commit-url-check.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb index 72c948f..7cac0d5 100755 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -21,12 +21,12 @@ class GiteeCommitUrlCheck end def gitee_commit_check - repo_dir = @url.split('/')[-3] repo_url = [@base_url, 'git'].join('.') + repo_dir = repo_url.split('/')[-1] commit_id = @url.split('/')[-1] Dir.chdir '/tmp' - %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + %x(/usr/bin/git clone --bare #{repo_url} #{repo_dir}) email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] -- 2.23.0

1 0

[PATCH compass-ci] doc/job/: add os_fields.md
by Zhang Yu 25 Nov '20

25 Nov '20

Add a document for introducing the os, os_version, os_arch fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os_fields.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/job/os_fields.md diff --git a/doc/job/os_fields.md b/doc/job/os_fields.md new file mode 100644 index 0000000..1e6baae --- /dev/null +++ b/doc/job/os_fields.md @@ -0,0 +1,15 @@ +# os, os_version, os_arch + +Meaning: +- Each test tells which os it needs, and the os_version, os_arch is also need to be specified, then the machine will run an os according to those three fields. +- Os, os_version, os_arch is the key for users to specify the os-related parameters. +- If os, os_version, os_arch is not given by users, it will use the default openeuler os-related parameters. +- Here are the os, os_version, os_arch that we support: + os os_version os_arch + openeuler 20.03, 1.0 aarch64, x86_64 + debian sid, 10 aarch64, x86_64 + centos 7.6, 7.8 aarch64, x86_64 + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 -- 2.23.0

4 6

[PATCH compass-ci] block_helper.cr: change if block into one line
by Xu Xijian 25 Nov '20

25 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/block_helper.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/lib/block_helper.cr b/src/lib/block_helper.cr index 37ad367..596ef7e 100644 --- a/src/lib/block_helper.cr +++ b/src/lib/block_helper.cr @@ -28,9 +28,7 @@ class BlockHelper @block_helper[uuid] = fiber end - if yield == true - spawn fiber.run - end + spawn fiber.run if yield == true until fiber.dead? Fiber.yield -- 2.23.0

2 2

[PATCH compass-ci] lib/compare_matrixes.rb: fix rubocop offenses
by Zhang Yu 25 Nov '20

25 Nov '20

Inspecting 1 file Offenses: compare_matrixes.rb:469:5: C: Layout/EmptyLineAfterGuardClause: Add empty line after guard clause. next if values['average'].size < 2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/compare_matrixes.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 0b49e59..1550435 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -467,6 +467,7 @@ end def assign_metric_change(metrics_values) metrics_values.each do |metric, values| next if values['average'].size < 2 + metrics_values[metric]['change'] = {} dimension_list = values['average'].keys -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 4 ++ .../template/template-defaults.yaml | 71 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 71 +++++++++++++++++++ spec/submit_spec.rb | 4 ++ 5 files changed, 151 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/template/template-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..6d6fb37c --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,4 @@ +suite: test +category: benchmark + +test: diff --git a/spec/submit/merge_yaml/template/template-defaults.yaml b/spec/submit/merge_yaml/template/template-defaults.yaml new file mode 100644 index 00000000..f1d13d04 --- /dev/null +++ b/spec/submit/merge_yaml/template/template-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +test: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..f1d13d04 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +test: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH v3 lkp-tests] daemon/sshd: fix pub_key to my_ssh_pubkey
by Zhang Yale 25 Nov '20

25 Nov '20

[why] 1. "my_ssh_pubkey" field is a collection of all pub_keys submitted by users. 2. We can save all the pub_keys submitted by users in the ES database and password-free login. 3. Therefore, users can sometimes borrow a machine to test without providing a pub_key. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index a3fdf954..c7d7b60c 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,5 +1,5 @@ #!/bin/sh -# pub_key +# my_ssh_pubkey # my_email # sshr_port # sshr_port_len @@ -14,10 +14,10 @@ sshr_ip=$LKP_SERVER run_ssh() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh - echo "$pub_key" > /root/.ssh/authorized_keys + echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys systemctl start sshd } @@ -73,11 +73,11 @@ compose_email() set_port() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return for i in $(seq 1 10) do - port=$(($(date +%s%N)%${sshr_port_len}+${sshr_port_base})) + port=$(($(date +%s%N)%$sshr_port_len+$sshr_port_base)) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return -- 2.23.0

2 2

[PATCH v2 lkp-tests] lkp-tests/*: update AUR Helper for archlinux
by Wang Yong 25 Nov '20

25 Nov '20

yaourt is unmaintained and yay also support native pacman operation, so only minor changes are needed Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 15 +++++++++++++++ distro/installer/archlinux | 15 +++++++++------ lib/install.sh | 2 +- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/distro/archlinux b/distro/archlinux index 408c5fa23..3f8d62ebc 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -109,3 +109,18 @@ pack_benchmark() ) done } + +add_aur_helper() +{ + echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + pacman -Q yay 2>/dev/null && return + + local user=lkp + local dir=/home/${user} + + mkdir -p ${dir} && chown ${user}:${user} ${dir} + pacman -S --needed --noconfirm git base-devel 2>/dev/null + su - ${user} -c "git clone https://aur.archlinux.org/yay.git" + su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" +} diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 27cdc63cf..af98e466f 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -2,12 +2,15 @@ # Archlinux package installation -# yaourt is a wrapper for pacman that adds automated access -# to the AUR. -# See: https://wiki.archlinux.org/index.php/Yaourt +# Yet Another Yogurt - An AUR Helper Written in Go +# Yay is based on the design of yaourt, apacman and pacaur +# See: https://github.com/Jguer/yay -# yaourt not support root to build package +# Yay not support root to build package # Use regular user to do this -regular_user=`grep 1000 /etc/passwd | awk -F':' '{ print $1 }'` +. $LKP_SRC/distro/${DISTRO} +add_aur_helper -sudo -u $regular_user yaourt -S --noconfirm $* +regular_user=`grep 1090 /etc/passwd | awk -F':' '{ print $1 }'` + +sudo -u $regular_user yay -S --needed --noconfirm $* diff --git a/lib/install.sh b/lib/install.sh index 881ade9f4..bf7bf21d7 100755 --- a/lib/install.sh +++ b/lib/install.sh @@ -17,7 +17,7 @@ sync_distro_sources() yum update fi ;; redhat) yum update ;; - archlinux) yaourt -Sy ;; + archlinux) yay -Sy ;; opensuse) zypper update ;; oracle) yum update ;; -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 22 +++++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 28 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 110 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..8564d443 --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,22 @@ +suite: ftq +category: noise-benchmark + +nr_task: 100% + +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- + +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..855e7210 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..78725424 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..40edfcb2 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ce6fed25 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

2 1

[PATCH v3 compass-ci 2/2] job.cr: give kernel_version a default value
by Xu Xijian 25 Nov '20

25 Nov '20

If the kernel_version is not set by users, give it a default value as same as vmlinuz version. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b533665..b87b55f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -358,11 +358,11 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? - self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["kernel_version"] ||= File.basename(File.real_path "#{boot_dir}/vmlinuz").lchop("vmlinuz-") + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz-#{kernel_version}") if "#{os_mount}" == "initramfs" - self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") - self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules-#{kernel_version}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers-#{kernel_version}.cgz") end end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] block_helper.cr: change if block into one line
by Xu Xijian 25 Nov '20

25 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/block_helper.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/lib/block_helper.cr b/src/lib/block_helper.cr index 37ad367..596ef7e 100644 --- a/src/lib/block_helper.cr +++ b/src/lib/block_helper.cr @@ -28,9 +28,7 @@ class BlockHelper @block_helper[uuid] = fiber end - if yield == true - spawn fiber.run - end + spawn fiber.run if yield == true until fiber.dead? Fiber.yield -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 + .../template/template-defaults.yaml | 71 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 71 +++++++++++++++++++ spec/submit_spec.rb | 4 ++ 5 files changed, 150 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/template/template-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..60e3df16 --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: test +category: benchmark +test: diff --git a/spec/submit/merge_yaml/template/template-defaults.yaml b/spec/submit/merge_yaml/template/template-defaults.yaml new file mode 100644 index 00000000..f1d13d04 --- /dev/null +++ b/spec/submit/merge_yaml/template/template-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +test: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..f1d13d04 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +test: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

3 2

[PATCH compass-ci 2/2] sparrow/0-package: fix docker.service restart failed
by Liu Yinsi 25 Nov '20

25 Nov '20

[why] when execute systemctl restart docker, error message: localhost.localdomain systemd[1]:docker.service: Failed with result 'timeout'. localhost.localdomain systemd[1]:Failed to start Docker Application Container Engine. because docker restart default timeout is 90s, when run too much containers, restart docker may need several minutes, then restart failed. [how] set TimeoutSec=0 to avoid timeout. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 5eb9f76..2753b9d 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -40,4 +40,7 @@ cat > /etc/docker/daemon.json <<EOF } EOF +sed '/^Environment=/a\TimeoutSec=0' /usr/lib/systemd/system/docker.service + +systemctl daemon-reload systemctl restart docker -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow/0-package: fix failed to start ssh-r
by Liu Yinsi 25 Nov '20

25 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..5eb9f76 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,7 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 EOF sysctl -p -- 2.23.0

1 0

[PATCH compass-ci] job.cr: give kernel_version a default value
by Xu Xijian 25 Nov '20

25 Nov '20

If the kernel_version is not set by users, give it a default value as same as vmlinuz version. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b533665..fa804e5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -358,11 +358,11 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? - self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["kernel_version"] ||= File.basename(File.real_path "#{boot_dir}/vmlinuz").gsub("vmlinuz-", "") + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz-#{kernel_version}") if "#{os_mount}" == "initramfs" - self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") - self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules-#{kernel_version}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers-#{kernel_version}.cgz") end end -- 2.23.0

1 1

[PATCH v2 compass-ci] job.cr: give kernel_version a default value
by Xu Xijian 25 Nov '20

25 Nov '20

If the kernel_version is not set by users, give it a default value as same as vmlinuz version. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b533665..a9923cc 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -358,11 +358,11 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? - self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["kernel_version"] ||= File.basename(File.real_path "#{boot_dir}/vmlinuz").gsub("vmlinuz-", "") + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz-#{kernel_version}") if "#{os_mount}" == "initramfs" - self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") - self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules-#{kernel_version}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers-#{kernel_version}.cgz") end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] daemon/sshd: fix pub_key to my_ssh_pubkey
by Zhang Yale 25 Nov '20

25 Nov '20

[Why] - "my_ssh_pubkey" field is a set of all pub_keys that had been submitted by users. - We can save all the pub_keys submitted by users in the ES database and set the login mode to password-free. - Therefore, users can sometimes borrow a machine to test without providing a pub_key. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index a3fdf954..c7d7b60c 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,5 +1,5 @@ #!/bin/sh -# pub_key +# my_ssh_pubkey # my_email # sshr_port # sshr_port_len @@ -14,10 +14,10 @@ sshr_ip=$LKP_SERVER run_ssh() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh - echo "$pub_key" > /root/.ssh/authorized_keys + echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys systemctl start sshd } @@ -73,11 +73,11 @@ compose_email() set_port() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return for i in $(seq 1 10) do - port=$(($(date +%s%N)%${sshr_port_len}+${sshr_port_base})) + port=$(($(date +%s%N)%$sshr_port_len+$sshr_port_base)) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return -- 2.23.0

1 1

[PATCH lkp-tests] setup/sysctl: add parameters
by Zhang Yu 25 Nov '20

25 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- setup/sysctl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/sysctl b/setup/sysctl index 81374939..913b83d3 100755 --- a/setup/sysctl +++ b/setup/sysctl @@ -202,6 +202,9 @@ # - sc_netdev_max_backlog # - sc_rmem_max # - sc_wmem_max +# - sc_tcp_rmem +# - sc_tcp_wmem +# - sc_sched_features . $LKP_SRC/lib/reproduce-log.sh @@ -408,5 +411,8 @@ [ -n "$sc_netdev_max_backlog" ] && log_eval "echo '$sc_netdev_max_backlog' > '/proc/sys/net/core/netdev_max_backlog'" [ -n "$sc_rmem_max" ] && log_eval "echo '$sc_rmem_max' > '/proc/sys/net/core/rmem_max'" [ -n "$sc_wmem_max" ] && log_eval "echo '$sc_wmem_max' > '/proc/sys/net/core/wmem_max'" +[ -n "$sc_tcp_rmem" ] && log_eval "echo '$sc_tcp_rmem' > '/proc/sys/net/ipv4/tcp_rmem'" +[ -n "$sc_tcp_wmem" ] && log_eval "echo '$sc_tcp_wmem' > '/proc/sys/net/ipv4/tcp_wmem'" +[ -n "$sc_sched_features" ] && log_eval "echo '$sc_sched_features' > '/sys/kernel/debug/sched_features'" exit 0 -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: fix repeated setting job
by Wu Zhende 25 Nov '20

25 Nov '20

Only first time should full initialize the job. Next time function "initialized?" will return true, just to use job_content, don't need to set default values. But now, "kernel_version" will let "initialized?" return false. Fix this. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b533665..e56f248 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -67,6 +67,8 @@ class Job set_defaults() end + # Each key must exist after the job is initialized + # Otherwise "initialized?" will return false METHOD_KEYS = %w( id os @@ -95,7 +97,6 @@ class Job kernel_append_root kernel_params docker_image - kernel_version linux_vmlinuz_path linux_modules_initrd linux_headers_initrd @@ -358,7 +359,7 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? + suffix = "-#{self["kernel_version"]}" if self["kernel_version"]? self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") if "#{os_mount}" == "initramfs" self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] sbin/submit: add job into my queue
by Xiao Shenwei 25 Nov '20

25 Nov '20

add --my-queue parameter the job will be add into your own queue usage: submit --my-queue testbox=vm-2p8g ../jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/submit | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/sbin/submit b/sbin/submit index 9bba1018..be588bed 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,6 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} +opt_my_queue = false actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -54,6 +55,11 @@ options = OptionParser.new do |opts| ARGV.shift end end + + opts.on('--my-queue', 'add to my queue') do + opt_my_queue = true + end + end options.parse!(ARGV) @@ -93,6 +99,10 @@ def find_jobfile(jobfile) exit 1 end +def add_my_queue(job) + job['queue'] = "#{job['tbox_group']}~#{ENV['USER']}" +end + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -134,6 +144,7 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) + add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

2 1

[PATCH v2 compass-ci] doc/job/: add os_fields.md
by Zhang Yu 25 Nov '20

25 Nov '20

Add a document for introducing the os, os_version, os_arch fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os_fields.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/os_fields.md diff --git a/doc/job/os_fields.md b/doc/job/os_fields.md new file mode 100644 index 0000000..942f4b4 --- /dev/null +++ b/doc/job/os_fields.md @@ -0,0 +1,16 @@ +# os, os_version, os_arch + +Meaning: +- Each test tells which os it needs, and the os_version, os_arch are also need to be specified, + then the machine will run an os according to these three fields. +- These os, os_version, os_arch are the key for users to specify the os related parameters. +- If os, os_version, os_arch are not given by users, it will use the default openeuler os related parameters. +- Here are some examples: + os os_version os_arch + openeuler 20.03, 1.0 aarch64, x86_64 + debian sid, 10 aarch64, x86_64 + centos 7.6, 7.8 aarch64, x86_64 + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: add my_ssh_pubkey
by Wu Zhende 25 Nov '20

25 Nov '20

Because pub_key is in sshd, and sshd is a hash. Hash will skip in "sh_export_top_env", So job.sh can't get the array pub_key to do password-free login. Add a new key "my_ssh_pubkey" in job to solve this problem. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 785daea..b533665 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -206,6 +206,7 @@ class Job sshd["pub_key"] = @account_info["my_ssh_pubkey"] @hash["sshd"] = JSON.parse(sshd.to_json) + @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] end private def update_account_my_pub_key(pub_key) -- 2.23.0

1 0

[PATCH v2 compass-ci] add-software: add reminders
by Wang Chenglong 25 Nov '20

25 Nov '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/add-software/run | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/add-software/run b/container/add-software/run index 089efd2..62d4eff 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -9,6 +9,8 @@ 2> ./run /os/centos/aarch64/7.6.1810 $(show-depends-packages centos) centos is an adaption file contain packages mapping from debian to centos, the whole path is '$LKP_SRC/distro/adaptation/centos'. + [Remind] + This tool also works for openEuler. EOF exit } -- 2.23.0

1 0

[PATCH compass-ci] add-software: add reminders
by Wang Chenglong 25 Nov '20

25 Nov '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/add-software/run | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/add-software/run b/container/add-software/run index 089efd2..dcbb8f6 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -9,6 +9,8 @@ 2> ./run /os/centos/aarch64/7.6.1810 $(show-depends-packages centos) centos is an adaption file contain packages mapping from debian to centos, the whole path is '$LKP_SRC/distro/adaptation/centos'. + [Remind] + This tool also works for Openeuler. EOF exit } -- 2.23.0

2 2

[PATCH compass-ci] remove empty changes for metric compare result
by Lu Weitao 25 Nov '20

25 Nov '20

[Why] With template compare mode, it's useless return an empty changes for a metric like: { ... "tables": { "vmstat.system.cs": { ... "change": { "dimensions": [ "compare_dimension" ], "source": [ [ "8G", "16G" ] ] } } } [How] Skip assign metric change value, if the metric only have one dimension, the empty change will disappear automatically Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 14c14cb..0b49e59 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -466,8 +466,8 @@ end def assign_metric_change(metrics_values) metrics_values.each do |metric, values| - metrics_values[metric]['change'] = {} next if values['average'].size < 2 + metrics_values[metric]['change'] = {} dimension_list = values['average'].keys dimension_groups = get_dimensions_combination(dimension_list) -- 2.23.0

1 0

[PATCH compass-ci] doc/job/: add os.md
by Zhang Yu 25 Nov '20

25 Nov '20

add a document for introducing the os field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/job/os.md diff --git a/doc/job/os.md b/doc/job/os.md new file mode 100644 index 0000000..0d0102c --- /dev/null +++ b/doc/job/os.md @@ -0,0 +1,14 @@ +# os + +Meaning: +- Each test tells which os is to be testing on, the vm will run a os according to different need. +- Os is a key for users to specify a os type. +- If os is not given by users, it will use the default debian. +- Here are the os type that we support: + openeuler + debian + centos + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 os_mount=cifs -- 2.23.0

2 2

Re: [PATCH lkp-tests] stats/iperf: fix rubocop offenses
by Wei Jihui 25 Nov '20

25 Nov '20

hi, everyone, some lkp-tests's scripts is not our create, it is better not to modify them. Thanks, Jihui Reply-To: In-Reply-To: <20201125015157.2194483-1-2134782174(a)qq.com> On Wed, Nov 25, 2020 at 09:51:57AM +0800, Zhang Yu wrote: >Inspecting 1 file > >iperf:19:86: C: Style/IfUnlessModifier: Modifier form of if makes the line too long. >puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s if obj['end']['streams'][0]['sender'] > ^^ > >iperf:20:90: C: Style/IfUnlessModifier: Modifier form of if makes the line too long. >puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s if obj['end']['streams'][0]['receiver' > ^^ > >Signed-off-by: Zhang Yu <2134782174(a)qq.com> >--- > stats/iperf | 12 +++++++++--- > 1 file changed, 9 insertions(+), 3 deletions(-) > >diff --git a/stats/iperf b/stats/iperf >index b5c6c403..f2521db6 100755 >--- a/stats/iperf >+++ b/stats/iperf >@@ -16,6 +16,12 @@ end > > exit if obj['end'].empty? > >-puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s if obj['end']['streams'][0]['sender'] >-puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s if obj['end']['streams'][0]['receiver'] >-puts 'udp.bps: ' + obj['end']['streams'][0]['udp']['bits_per_second'].to_s if obj['end']['streams'][0]['udp'] >+if obj['end']['streams'][0]['sender'] >+ puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s >+end >+if obj['end']['streams'][0]['receiver'] >+ puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s >+end >+if obj['end']['streams'][0]['udp'] >+ puts 'udp.bps: ' + obj['end']['streams'][0]['udp']['bits_per_second'].to_s >+end >-- >2.23.0 >

1 0