- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sched: simplify cluster state updating
by Ren Wen 06 Nov '20

06 Nov '20

There are three steps when updating cluster state: 1) get cluster state from redis. 2) update cluster state. 3) rewrite to redis. Before: write to redis one job info once. After: write to redis more than one job info once. It will save time when writing more than one job info once. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4b12b4..bff6090 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -58,11 +58,11 @@ class Sched return cluster_state end - # get -> modify -> set - def update_cluster_state(cluster_id, job_id, property, value) + # Update job info according to cluster id. + def update_cluster_state(cluster_id, job_id, job_info : Hash(String, String)) cluster_state = get_cluster_state(cluster_id) if cluster_state[job_id]? - cluster_state[job_id].merge!({property => value}) + cluster_state[job_id].merge!(job_info) @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) end end @@ -86,9 +86,9 @@ class Sched case request_state when "abort", "finished", "failed" # update node state only - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) @block_helper.block_until_finished(cluster_id) { cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) cluster_state == "ready" || cluster_state == "abort" @@ -96,7 +96,7 @@ class Sched return cluster_state when "wait_finish" - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) @@ -110,10 +110,11 @@ class Sched direct_ips = env.params.query["direct_ips"] direct_macs = env.params.query["direct_macs"] - update_cluster_state(cluster_id, job_id, "roles", node_roles) - update_cluster_state(cluster_id, job_id, "ip", node_ip) - update_cluster_state(cluster_id, job_id, "direct_ips", direct_ips) - update_cluster_state(cluster_id, job_id, "direct_macs", direct_macs) + job_info = {"roles" => node_roles, + "ip" => node_ip, + "direct_ips" => direct_ips, + "direct_macs" => direct_macs} + update_cluster_state(cluster_id, job_id, job_info) when "roles_ip" role = "server" role_state = get_role_state(cluster_id, role) -- 2.23.0

3 4

[PATCH v6 lkp-tests] tests/build-pkg: change location for *.cgz file
by Liu Shaofei 06 Nov '20

06 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: - path: /srv/initrd/build-pkg/initramfs/openeuler/aarch64/20.03/adl-git/ - filename: . ├── 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz ├── df0584e46db9e7b412626deab10b923504745b2c.cgz ├── latest.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz ├── v1.0.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz └── v2.0.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz 0 directories, 5 files Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..ad8d4843 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,21 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + # the same image is mounted to cifs and nfs, the generated cgz files + # are stored in the nfs directory. + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +83,27 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 +} + +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" } check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

4 4

[PATCH v3 compass-ci 6/6] container/webhook: add scripts for webhook
by Li Yuanchao 06 Nov '20

06 Nov '20

webhook.js will listen to webhook message from 'gitee.com' or 'github.com' if catch 'push' message, call push_hook.rb to send message to git-mirror through message queue Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/root/push_hook.rb | 15 +++++++++++++++ container/webhook/root/webhook.js | 22 ++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/webhook/root/push_hook.rb create mode 100755 container/webhook/root/webhook.js diff --git a/container/webhook/root/push_hook.rb b/container/webhook/root/push_hook.rb new file mode 100755 index 0000000..e467ccd --- /dev/null +++ b/container/webhook/root/push_hook.rb @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +connection = Bunny.new('amqp://172.17.0.1:5672') +connection.start +channel = connection.create_channel + +queue = channel.queue('web_hook') +message = ARGV[0] +queue.publish(message) +connection.close diff --git a/container/webhook/root/webhook.js b/container/webhook/root/webhook.js new file mode 100755 index 0000000..a98576c --- /dev/null +++ b/container/webhook/root/webhook.js @@ -0,0 +1,22 @@ +const http = require('http') +const spawn = require('child_process').spawn +const createHandler = require('git-webhook-handler') +// if users want to use webhook, they need to config webhook on their code warehouse('gitee.com' or 'github.com'), +// and use url: http://183.134.196.212:11301/webhook, secret: webhook@git. +const handler = createHandler({ path: '/webhook', secret: 'webhook@git' }) + +handler.on('error', function(err){ + console.error('Error:', err.message) +}) + +handler.on('push', function(event){ + console.log(event.payload.repository.url) + spawn('ruby', ['push_hook.rb', event.payload.repository.url]) +}) + +http.createServer(function(req, res){ + handler(req, res, function(err){ + res.statusCode = 404 + res.end('no such location') + }) +}).listen(11301) -- 2.23.0

1 0

[PATCH v3 compass-ci 5/6] container/webhook: start-depends
by Li Yuanchao 06 Nov '20

06 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/webhook/start-depends diff --git a/container/webhook/start-depends b/container/webhook/start-depends new file mode 100755 index 0000000..5232abf --- /dev/null +++ b/container/webhook/start-depends @@ -0,0 +1 @@ +rabbitmq -- 2.23.0

1 0

[PATCH v3 compass-ci 4/6] container/webhook: start
by Li Yuanchao 06 Nov '20

06 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 container/webhook/start diff --git a/container/webhook/start b/container/webhook/start new file mode 100755 index 0000000..373396a --- /dev/null +++ b/container/webhook/start @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "webhook" + +cmd = %w[ + docker run + --restart=always + --name webhook + -d + -p 11301:11301 + node:webhook +] + +cmd += ['sh', '-c', 'node webhook.js'] +system(*cmd) -- 2.23.0

1 0

[PATCH v3 compass-ci 3/6] container/webhook: build
by Li Yuanchao 06 Nov '20

06 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/webhook/build diff --git a/container/webhook/build b/container/webhook/build new file mode 100755 index 0000000..bb5c0a7 --- /dev/null +++ b/container/webhook/build @@ -0,0 +1,6 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t node:webhook . + -- 2.23.0

1 0

[PATCH v3 compass-ci 2/6] container/webhook: Dockerfile
by Li Yuanchao 06 Nov '20

06 Nov '20

add container webhook Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/webhook/Dockerfile diff --git a/container/webhook/Dockerfile b/container/webhook/Dockerfile new file mode 100644 index 0000000..9299bb6 --- /dev/null +++ b/container/webhook/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM node:10.21.0-alpine +MAINTAINER Li Yuanchao <lyc163mail(a)163.com> + +ADD root / + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk upgrade && \ + apk add --no-cache git && \ + apk add npm ruby-dev + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + +RUN npm install git-webhook-handler && \ + gem install bunny + -- 2.23.0

1 0

[PATCH v3 compass-ci 1/6] git_mirror: add webhook handler
by Li Yuanchao 06 Nov '20

06 Nov '20

webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..9d6a45d 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end def fork_stat_init(stat_key) @@ -287,3 +293,39 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + + # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + fork_name, project = webhook_url.split('/')[-2, 2] + + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + puts "webhook: #{webhook_url} is not found!" + end + + def handle_webhook + Thread.new do + @webhook_queue.subscribe(block: true) do |_delivery, _properties, webhook_url| + git_repo = get_git_repo(webhook_url) + do_push(git_repo) if git_repo + sleep(0.1) + end + end + end +end -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 06 Nov '20

06 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..cd133d9e --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,37 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + if [ -x /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + else + echo "The hackbench executable file cannot be found." && exit 1 + fi + + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

2 4

[PATCH v1 lab-z9] cluster: add cluster config file
by Zhang Yu 06 Nov '20

06 Nov '20

About config filename: "cs-s1-a102-c1": - cs : cluster - s1 : one server node - a102 : server node name - c1 : one client node Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- cluster/cs-s1-a102-c1 | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 cluster/cs-s1-a102-c1 diff --git a/cluster/cs-s1-a102-c1 b/cluster/cs-s1-a102-c1 new file mode 100644 index 0000000..af57123 --- /dev/null +++ b/cluster/cs-s1-a102-c1 @@ -0,0 +1,10 @@ +switch: Switch-P10 +ip0: 1 +nodes: + taishan200-2280-2s48p-512g--a102: + roles: [ server ] + macs: [ "44:67:47:c9:ea:37" ] + + taishan200-2280-2s48p-256g--a101: + roles: [ client ] + macs: [ "44:67:47:d7:6c:a3" ] -- 2.23.0

1 0