- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/3] git_mirror: put repo info to es
by Li Yuanchao 16 Oct '20

16 Oct '20

move a method to pass rubocop limit: Class xxx has too many lines[101/100] Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5f7fa6b..fef4aca 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -9,6 +9,8 @@ require 'json' # gem install PriorityQueue require 'priority_queue' require 'English' +require 'elasticsearch' +require_relative 'constants.rb' # worker threads class GitMirror @@ -84,6 +86,7 @@ class MirrorMain @git_info = {} @defaults = {} @git_queue = Queue.new + @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info connection = Bunny.new('amqp://172.17.0.1:5672') connection.start @@ -108,16 +111,6 @@ class MirrorMain @defaults[project] = YAML.safe_load(File.open(defaults_file)) end - def load_repo_file(repodir, project, fork_name) - git_repo = "#{project}/#{fork_name}" - @git_info[git_repo] = YAML.safe_load(File.open(repodir)) - @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] - fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 - end - def traverse_repodir(repodir) if File.directory? repodir load_defaults(repodir) @@ -194,6 +187,17 @@ end # main thread class MirrorMain + def load_repo_file(repodir, project, fork_name) + git_repo = "#{project}/#{fork_name}" + @git_info[git_repo] = YAML.safe_load(File.open(repodir)) + @git_info[git_repo]['git_repo'] = git_repo + @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] + es_repo_update(git_repo, @git_info[git_repo]) + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority + @priority += 1 + end + def compare_refs(cur_refs, old_refs) new_refs = { heads: {} } cur_refs[:heads].each do |ref, commit_id| @@ -246,4 +250,12 @@ class MirrorMain load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) end end + + def es_repo_update(git_repo, repo_info) + body = { + "doc": repo_info, + "doc_as_upsert": true + } + @es_client.update(index: 'repo', type: '_doc', id: git_repo, body: body) + end end -- 2.23.0

2 1

[PATCH lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 16 Oct '20

16 Oct '20

If ipmitool returns error or empty, the return of "$(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" is null, and then only add the field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/host-info b/tests/host-info index a57855dc..cd77c1eb 100755 --- a/tests/host-info +++ b/tests/host-info @@ -69,8 +69,7 @@ get_partitions() # sdf 1 ssd=$(lsblk -d -o name,ROTA |grep 0$ | awk '{print $1}') hdd=$(lsblk -d -o name,ROTA |grep 1$ | awk '{print $1}') - if [ -n "$hdd" ]; then - echo "hdd_partitions:" + if [ -n "$hdd" ]; then echo "hdd_partitions:" get_part $hdd fi @@ -80,6 +79,11 @@ get_partitions() fi } +get_ipmi_ip() +{ + echo "ipmi_ip: $(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" +} + main() { get_node @@ -87,6 +91,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

1 1

[PATCH compass-ci] fix docker-pull can't be used
by Xiao Shenwei 16 Oct '20

16 Oct '20

when run multi-docker job, the PATH exclude $CCI_SRC, so docker-pull can't be used directly, use full path to replace Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index b454f68..c241d3c 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -69,7 +69,7 @@ end def run(hostname, load_path, hash) docker_image = hash['docker_image'] - system "docker-pull #{docker_image}" + system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, ENV['CCI_SRC'] + '/providers/docker/run.sh' -- 2.23.0

1 0

[PATCH compass-ci] sched: throw an exception when deps/pkg does not exist
by Cao Xueliang 16 Oct '20

16 Oct '20

When submitting a job, an exception is thrown when deps/pkg does not exist on the server. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..ef515b8 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -389,11 +389,21 @@ class Job program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) - initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?(deps_dest_file) + initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) + else + raise "#{deps_dest_file} does not exist on the server." + end end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") && File.exists?(pkg_dest_file) - initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") + if File.exists?(pkg_dest_file) + initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) + else + raise "#{pkg_dest_file} does not exist on the server." + end end end end -- 2.23.0

1 1

[PATCH lkp-tests] lib/job.rb: load host config with atomic job
by Wei Jihui 15 Oct '20

15 Oct '20

according to commit c725e0f022b707ba385f8f3657e79, job = deepcopy(@job) without load_hosts_config it will keep to @job for "@job = deepcopy(job)" under "@jobs.each do|hash|" so this @job is without load_hosts_config. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..6359254e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -696,9 +696,11 @@ class Job each_job_init each_job(&block) @jobs.each do |hash| + @load_hosts_done = false @job = deepcopy(job) @job2 = hash load_defaults + load_hosts_config each_job_init each_job(&block) end -- 2.23.0

2 1

[PATCH compass-ci 3/3] container: git-mirror: add elasticsearch in dockerfile
by Li Yuanchao 15 Oct '20

15 Oct '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 9c582f7..390bbcc 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -12,5 +12,5 @@ RUN apk update && \ apk add ruby-dev make gcc g++ RUN umask 002 && \ - gem install bunny json PriorityQueue --no-rdoc --no-ri + gem install bunny json PriorityQueue elasticsearch --no-rdoc --no-ri -- 2.23.0

1 0

[PATCH compass-ci 1/3] git_mirror: optimize variable name
by Li Yuanchao 15 Oct '20

15 Oct '20

the same thing, the same name Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 779b488..5f7fa6b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -54,7 +54,7 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['forkdir']}.git" + mirror_dir = "/srv/git/#{fork_info['git_repo']}.git" possible_new_refs = false if File.directory?(mirror_dir) possible_new_refs = git_fetch(mirror_dir) @@ -62,7 +62,7 @@ class GitMirror FileUtils.mkdir_p(mirror_dir) possible_new_refs = git_clone(fork_info['url'], mirror_dir) end - feedback(fork_info['forkdir'], possible_new_refs) + feedback(fork_info['git_repo'], possible_new_refs) end def git_mirror @@ -109,11 +109,12 @@ class MirrorMain end def load_repo_file(repodir, project, fork_name) - @git_info["#{project}/#{fork_name}"] = YAML.safe_load(File.open(repodir)) - @git_info["#{project}/#{fork_name}"]['forkdir'] = "#{project}/#{fork_name}" - @git_info["#{project}/#{fork_name}"].merge!(@defaults[project]) if @defaults[project] - fork_stat_init("#{project}/#{fork_name}") - @priority_queue.push "#{project}/#{fork_name}", @priority + git_repo = "#{project}/#{fork_name}" + @git_info[git_repo] = YAML.safe_load(File.open(repodir)) + @git_info[git_repo]['git_repo'] = git_repo + @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority @priority += 1 end -- 2.23.0

1 0

[PATCH lkp-tests] spec: update submit_spec.rb results
by Wei Jihui 15 Oct '20

15 Oct '20

run rspec spec/submit_spec.rb the results is change. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/ftq/ftq-1000-2-100000ss-add.yaml | 4 ++-- spec/submit/ftq/ftq-10000-2-100000ss-add.yaml | 4 ++-- spec/submit/ftq/ftq-20-2-6000ss-cache.yaml | 4 ++-- ...4-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml index 7522fb96..3849bb30 100644 --- a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml +++ b/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml @@ -9,13 +9,13 @@ ftq: freq: 1000 testbox: vm-2p8g--spec_submit tbox_group: vm-2p8g--spec_submit +lab: spec_lab provider: qemu template: kvm -nr_cpu: 2 nr_node: 1 +nr_cpu: 2 memory: 8G arch: aarch64 -lab: spec_lab cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y diff --git a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml index ff941dfb..2d9785e4 100644 --- a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml +++ b/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml @@ -9,13 +9,13 @@ ftq: freq: 10000 testbox: vm-2p8g--spec_submit tbox_group: vm-2p8g--spec_submit +lab: spec_lab provider: qemu template: kvm -nr_cpu: 2 nr_node: 1 +nr_cpu: 2 memory: 8G arch: aarch64 -lab: spec_lab cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y diff --git a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml index 3fad02dd..6c915e93 100644 --- a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml +++ b/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml @@ -9,13 +9,13 @@ ftq: freq: 20 testbox: vm-2p8g--spec_submit tbox_group: vm-2p8g--spec_submit +lab: spec_lab provider: qemu template: kvm -nr_cpu: 2 nr_node: 1 +nr_cpu: 2 memory: 8G arch: aarch64 -lab: spec_lab cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y diff --git a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 5b3cb6ec..43132b8f 100644 --- a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -28,13 +28,13 @@ if role client: testbox: vm-2p8g--spec_submit tbox_group: vm-2p8g--spec_submit node_roles: server client +lab: spec_lab provider: qemu template: kvm -nr_cpu: 2 nr_node: 1 +nr_cpu: 2 memory: 8G arch: aarch64 -lab: spec_lab kmsg: boot-time: uptime: -- 2.23.0

1 0

[PATCH v7 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 15 Oct '20

15 Oct '20

Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 89 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..75802710 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key # email +# uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5051 +[ -n "$sshr_port_len" ] || sshr_port_len=3 +[ -n "$sshr_port_base" ] || sshr_port_base=30010 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $user_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: + login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch - Login: - ssh root@$PUB_IP - Due time: - $deadline +Regards +Compass-ci +" +} - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $user_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$uuid" ] || return 0 + user_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +get_port +report_ssh_port $port run_email -- 2.23.0

1 1

[PATCH compass-ci] fix: taskqueue_api: active close http client
by Tong Qunfeng 15 Oct '20

15 Oct '20

before patch: there left a lot of netstat in TIME_WAIT docker0_ip:3060 scheduler_ip:any_port TIME_WAIT it's timewait timer is 60s. after 60s, this any_port will be released after this patch: http connect to taskqueue will be release as soon as the visit is finished (after submit job | consume job) Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/taskqueue_api.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/taskqueue_api.cr b/src/lib/taskqueue_api.cr index 8d0db5d..99c3933 100644 --- a/src/lib/taskqueue_api.cr +++ b/src/lib/taskqueue_api.cr @@ -12,7 +12,9 @@ class TaskQueueAPI def add_task(service_queue_path : String, task : JSON::Any) params = HTTP::Params.encode({"queue" => service_queue_path}) - response = HTTP::Client.post("http://#{@host}:#{@port}/add?" + params, body: task.to_json) + client = HTTP::Client.new(@host, port: @port) + response = client.post("/add?" + params, body: task.to_json) + client.close arrange_response(response) end @@ -32,7 +34,9 @@ class TaskQueueAPI end private def response_put_api(cmd : String, params : String) - response = HTTP::Client.put("http://#{@host}:#{@port}/#{cmd}?" + params) + client = HTTP::Client.new(@host, port: @port) + response = client.put("/#{cmd}?" + params) + client.close arrange_response(response) end -- 2.23.0

1 0