December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sched.cr: put testbox state to es
by Li Yuanchao 16 Dec '20

16 Dec '20

History of testbox would be kept. [sample] { "_index" : "testbox", "_type" : "_doc", "_id" : "vm-2p16g.liyuanchao-969257", "_score" : 1.0, "_source" : { "state" : "rebooting", "job_id" : "crystal.190886", "history" : [ { "time" : "2020-12-15 19:52:14", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "running" }, { "time" : "2020-12-15 19:57:22", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "rebooting" } ] } } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/elasticsearch_client.cr | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 10a51ab..7cb0dbf 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -104,6 +104,7 @@ class Sched end @redis.update_wtmp(testbox, hash) + @es.update_tbox(testbox, hash) # json log hash["testbox"] = testbox diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 7c92204..b17d7f8 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -87,6 +87,27 @@ class Elasticsearch::Client ) end + def update_tbox(testbox : String, wtmp_hash : Hash) + query = {:index => "testbox", :type => "_doc", :id => testbox} + result = @client.get_source(query) if @client.exists(query) + history = result["history"].as_a? if result.is_a?(JSON::Any) + history ||= [] of JSON::Any + history << JSON.parse(wtmp_hash.to_json) + + @client.create( + { + :index => "testbox", + :type => "_doc", + :id => testbox, + :body => { + :state => wtmp_hash["state"], + :job_id => wtmp_hash["job_id"], + :history => history + } + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

2 1

[PATCH compass-ci] update-os-docker: add support sshd to docker images
by Xu Xijian 16 Dec '20

16 Dec '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/update-os-docker/sbin/setup | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/container/update-os-docker/sbin/setup b/container/update-os-docker/sbin/setup index a38e2df..e054a29 100755 --- a/container/update-os-docker/sbin/setup +++ b/container/update-os-docker/sbin/setup @@ -4,3 +4,9 @@ yum update -y yum install -y --skip-broken "$@" + +command -v sshd || yum install -y openssh-server +echo -e "y\n" | ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key +echo -e "y\n" | ssh-keygen -t rsa -f /etc/ssh/ssh_host_ecdsa_key +echo -e "y\n" | ssh-keygen -t rsa -f /etc/ssh/ssh_host_ed25519_key + -- 2.23.0

3 2

Re: [PATCH lkp-tests] fix: extra empty line
by Ren Wen 16 Dec '20

16 Dec '20

>oh, sorry i've delete it in next patch version Well, do it. Thanks, RenWen > >Thanks, >Zhangyu > >> } >> >> main() >>-- >>2.23.0 >>

1 0

[PATCH v2 compass-ci] assign-account: generate ssh keys for user by default
by Luan Shengde 16 Dec '20

16 Dec '20

[why] request is: when user submit jobs, the ssh keys are already there. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 10 ++-------- container/assign-account/get_account_info.rb | 4 +--- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 766e1b3..21285e7 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -61,7 +61,6 @@ stdin_info = {} email_info = {} my_info_es = {} conf_info = { - 'gen_sshkey' => false, 'enable_login' => true, 'is_update_account' => false } @@ -83,13 +82,12 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' - opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" opts.banner += " -n is required when assigning account with -e\n" opts.banner += " -s is optional when use -e\n" - opts.banner += " -g is optional, used to generate sshkey for user\n" opts.banner += " -u is required when updating an account\n" opts.banner += ' -l is optional, used to enable/disable login permission' @@ -141,10 +139,6 @@ options = OptionParser.new do |opts| init_info(mail_content, email_info, my_info, my_info_es) end - opts.on('-g', '--gen-sshkey', 'generate jumper rsa public/private key and return pubkey') do - conf_info['gen_sshkey'] = true - end - opts.on('-u', '--update', 'updata configurations') do conf_info['is_update_account'] = true end diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 9a7acda..67c4c46 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -112,9 +112,7 @@ class AccountStorage config_authorized_key(login_name) config_yaml(login_name) permit_login_config(login_name) - if @data.key?('gen_sshkey') && @data['gen_sshkey'] - my_jumper_pubkey = generate_ssh_key(login_name) - end + my_jumper_pubkey = generate_ssh_key(login_name) jumper_account_info = { 'my_login_name' => login_name, -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib: add common deps packages
by Zhang Yu 16 Dec '20

16 Dec '20

[why] when run job with initramfs, some command like time and perf are not found [how] add depends package of initramfs-common to install the common package Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8ce8649..31c4887 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -498,6 +498,7 @@ class Job private def get_depends_initrd(program_params, initrd_deps_arr, initrd_pkg_arr) initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + program_params["initramfs-common"] = JSON::Any.new("") if os_mount == "initramfs" program_params.keys.each do |program| if program =~ /^(.*)-\d+$/ -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: verify directory upload from redis
by Cui Yili 16 Dec '20

16 Dec '20

The previous verification file upload mechanism has poor scalability. change to get the upload directory from redis to verify better extension. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 76 ++++++++++++++++-------------- 1 file changed, 41 insertions(+), 35 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1ccaa92..af27653 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -4,6 +4,8 @@ user lkp; worker_processes auto; pid /run/nginx.pid; +env REDIS_HOST; +env REDIS_PORT; events {} @@ -16,7 +18,7 @@ http { access_log /tmp/access.log; error_log /tmp/error.log; - location /result { + location / { allow all; root /srv/; @@ -26,49 +28,53 @@ http { dav_access user:rw group:rw all:rw; access_by_lua_block { - accesskey = ngx.var.cookie_ACCESSKEY - job_id = string.sub(accesskey, string.find(accesskey,"-") - string.len(accesskey)) - uri = ngx.var.request_uri - path = string.match(uri,"%g*/") - root_path = string.sub(path, 1, string.find(path, job_id) - 1)..job_id.."/" + function split(str, reps) + local resultstrlist = {} + string.gsub(str,'[^'..reps..']+',function (w) + if w ~= nil then + table.insert(resultstrlist,w) + end + end) + return resultstrlist + end - accesskey_file = "/srv/"..root_path.."."..accesskey + redis_host = os.getenv("REDIS_HOST") + redis_port = os.getenv("REDIS_PORT") - local f = io.open(accesskey_file) - if not f then + local redis = require "resty.redis" + local red = redis:new() + red:set_timeout(1000) + local ok, err = red:connect(redis_host, redis_port) + if not ok then + ngx.log(ngx.ERR, "connect redis: ", err) ngx.exit(ngx.HTTP_FORBIDDEN) - end - io.close(f) - } - } - location /initrd { - allow all; - root /srv/; - autoindex on; - create_full_put_path on; - dav_methods PUT MKCOL; - dav_access user:rw group:rw all:rw; + end - access_by_lua_block { + job_id = ngx.var.cookie_JOBID uri = ngx.var.request_uri + path = string.match(uri, "%g*/") + upload_file = string.match(uri, "([^/]+)$") - path = string.match(uri,"%g*/") - upload_file = string.match(uri, ".+/([^/]*%.%w+)$") - second_dir = string.match(uri, "/*%w+/(.-)/.+") + local res, err = red:hget("sched/id2upload_dirs", job_id) + if not res or res == ngx.null then + ngx.log(ngx.ERR, "failed to hget: ", err) + ngx.exit(ngx.HTTP_FORBIDDEN) + end - if (second_dir == "pkg" or second_dir == "build-pkg") - then - link_name = "latest" - else - link_name = string.match(upload_file, "(.*)_%d+%.cgz") + if string.find(res, ",") then + table = split(res,",") + for i = 1, #table do + if string.find(path, table[i]) then + return + end end - - if (link_name ~= '') - then - link_name = link_name..".cgz" - root_patch = "/srv/"..path - io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) + else + if string.find(path, res, 1, true) then + return + end end + + ngx.exit(ngx.HTTP_FORBIDDEN) } } -- 2.23.0

1 0

[PATCH v2 lkp-tests] distro/depends: add initramfs-common
by Zhang Yu 16 Dec '20

16 Dec '20

When run job with initramfs, some dependency packages are required Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/initramfs-common | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/initramfs-common diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common new file mode 100644 index 000000000..5d3ecb76d --- /dev/null +++ b/distro/depends/initramfs-common @@ -0,0 +1,3 @@ +time +perf +which -- 2.23.0

1 0

[PATCH v4 compass-ci 6/6] fetch-mail: start file
by Luan Shengde 16 Dec '20

16 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/start | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/fetch-mail/start diff --git a/container/fetch-mail/start b/container/fetch-mail/start new file mode 100755 index 0000000..98cbd8f --- /dev/null +++ b/container/fetch-mail/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'fetch-mail' + +names = Set.new %w[ + FETCH_MAIL_ADDRESS + FETCH_MAIL_AUTH_CODE + FETCH_MAIL_DIR + FETCH_MAIL_BOX +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + -u mailer + --name=fetch-mail + -d +] + env + %W[ + -v #{defaults['FETCH_MAIL_DIR']}:#{defaults['FETCH_MAIL_DIR']}:rw + -w /home/mailer/ + fetch-mail +] + +cmd += ['sh', '-c', './run.sh'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v4 compass-ci 5/6] fetch-mail: build file
by Luan Shengde 16 Dec '20

16 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/fetch-mail/build diff --git a/container/fetch-mail/build b/container/fetch-mail/build new file mode 100755 index 0000000..e72bed8 --- /dev/null +++ b/container/fetch-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t fetch-mail:latest . -- 2.23.0

1 0

[PATCH v4 compass-ci 4/6] fetch-mail: Dockerfile
by Luan Shengde 16 Dec '20

16 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/Dockerfile | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 container/fetch-mail/Dockerfile diff --git a/container/fetch-mail/Dockerfile b/container/fetch-mail/Dockerfile new file mode 100644 index 0000000..5bd1a33 --- /dev/null +++ b/container/fetch-mail/Dockerfile @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM centos:latest +MAINTAINER Luan Shengde <shdluan(a)163.com> +ENV DEBIAN_FRONTEND noninteractive + +RUN useradd -u 2002 mailer +RUN yum update -y +RUN yum install -y procmail fetchmail + +COPY fetchmailrc.template /home/mailer/.fetchmailrc +COPY procmailrc.template /home/mailer/.procmailrc +COPY run.sh /home/mailer/ + +RUN chmod 600 /home/mailer/.fetchmailrc +RUN chmod 600 /home/mailer/.procmailrc +RUN chown -R mailer:mailer /home/mailer +RUN chmod +x /home/mailer/run.sh -- 2.23.0

1 0