December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v4 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 30 Dec '20

30 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..801649cf4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || return + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

2 2

[PATCH lkp-tests] makepkg.conf: use the my_curl command to use the cache
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- etc/makepkg.conf | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 1fffdc2d2..5d73399d9 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -8,10 +8,10 @@ # #-- The download utilities that makepkg should use to acquire sources # Format: 'protocol::agent' -DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u' - 'ftp::/usr/bin/curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' - 'http::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'https::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' +DLAGENTS=('file::/usr/bin/my_curl -gqC - -o %o %u' + 'ftp::/usr/bin/my_curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' + 'http::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'https::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'rsync::/usr/bin/rsync --no-motd -z %u %o' 'scp::/usr/bin/scp -C %u %o') -- 2.23.0

2 1

[PATCH compass-ci] delimiter/utils: set job testbox
by Cao Xueliang 29 Dec '20

29 Dec '20

If job has config field, set the testbox to "vm-2p32g", otherwise set the testbox to "vm-2p16g". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 17128ab..49d48ee 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -135,12 +135,14 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id + job['testbox'] = job['config'] ? 'vm-2p32g' : 'vm-2p16g' job.delete('error_ids') job.delete('start_time') job.delete('job_state') job.delete('end_time') job.delete('loadavg') + job.delete('queue') job.delete('stats') job.delete('id') -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: try to install ntpdate if there is no such pkg
by Yu Chuan 29 Dec '20

29 Dec '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ae90220750db..7d15bb370fa9 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -6,6 +6,7 @@ . $LKP_SRC/lib/reboot.sh . $LKP_SRC/lib/ucode.sh . $LKP_SRC/lib/tbox.sh +. $LKP_SRC/lib/detect-system.sh # borrowed from linux/tools/testing/selftests/rcutorture/doc/initrd.txt # Author: Paul E. McKenney <paulmck(a)linux.vnet.ibm.com> @@ -236,7 +237,13 @@ run_ntpdate() [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return - [ -x '/usr/sbin/ntpdate' ] || return + has_cmd ntpdate || { + detect_system + local DISTRO=$_system_name_lowercase + . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) + } + + has_cmd ntpdate || return local hour="$(date +%H)" -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/2] my_curl: change the curl request so that the cache can be used
by Cui Yili 29 Dec '20

29 Dec '20

change curl request, in order to use local cache. e.g. https://download.redis.io/releases/redis-4.0.2.tar.gz ==> https://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k http://download.redis.io/releases/redis-4.0.2.tar.gz ==> http://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k Signed-off-by: cuiyili <2268260388(a)qq.com> --- rootfs/addon/usr/bin/my_curl | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 rootfs/addon/usr/bin/my_curl diff --git a/rootfs/addon/usr/bin/my_curl b/rootfs/addon/usr/bin/my_curl new file mode 100755 index 000000000..97d5352a4 --- /dev/null +++ b/rootfs/addon/usr/bin/my_curl @@ -0,0 +1,22 @@ +#!/bin/bash + +declare -a array +i=0 + +if [ -n "$HTTP_PROXY_HOST" ]; then + for param in "$@"; + do + if [[ "$param" =~ http://(.*) ]] && [ -n "$HTTP_PROXY_PORT" ]; then + array[$i]="http://$HTTP_PROXY_HOST:$HTTP_PROXY_PORT/${BASH_REMATCH[1]}" + elif [[ "$param" =~ https://(.*) ]] && [ -n "$HTTPS_PROXY_PORT" ]; then + array[$i]="https://$HTTP_PROXY_HOST:$HTTPS_PROXY_PORT/${BASH_REMATCH[1]}" + else + array[$i]="$param" + fi + ((i++)) + done + + command curl -k ${array[*]} +else + command curl "$@" +fi -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] lkp-setup-rootfs: all use of env http_proxy and delete specific proxy settings
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- bin/lkp-setup-rootfs | 9 --------- 1 file changed, 9 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..a7f1a7ffb 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -105,7 +105,6 @@ install_proxy_yum() { [ -d "/etc/yum/" ] || return - echo "proxy=http://$SQUID_HOST:$SQUID_PORT" >> /etc/yum.conf if ls /etc/yum.repos.d/*.repo >/dev/null 2>&1; then sed -i 's/https:/http:/g' /etc/yum.repos.d/*.repo fi @@ -117,11 +116,6 @@ install_proxy_apt() { [ -d "/etc/apt/" ] || return - cat << EOF > /etc/apt/apt.conf.d/proxy -Acquire::http::proxy "http://$SQUID_HOST:$SQUID_PORT/"; -Acquire::ftp::proxy "ftp://$SQUID_HOST:$SQUID_PORT/"; -EOF - for cfg in /etc/apt/sources.list /etc/apt/sources.list.d/* do [ -f "$cfg" ] || continue @@ -176,9 +170,6 @@ install_proxy() install_goproxy - [ -z "$SQUID_HOST" ] && return - [ -z "$SQUID_PORT" ] && return - install_proxy_apt || install_proxy_yum } -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils: add query job stats multiple times
by Cao Xueliang 29 Dec '20

29 Dec '20

Sometimes, the job state is already extract_finished, but we can't query the job stats, so add a temporary solutions, query the job stats multiple times. The extract container will be improved in future. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 372e821..17128ab 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -83,10 +83,10 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? - es = ESQuery.new - new_job = es.query_by_id(new_job_id) + stats = query_stats(new_job_id, 10) + raise "es cant query #{new_job_id} stats field!" unless stats - status = new_job['stats'].key?(error_id) ? 'bad' : 'good' + status = stats.key?(error_id) ? 'bad' : 'good' puts "new_job_id: #{new_job_id}" puts "upstream_commit: #{job['upstream_commit']}" record_jobs(new_job_id, job['upstream_commit']) @@ -94,6 +94,21 @@ module Utils return status end + # sometimes the job_state is extract_finished + # but we cant query the job stats field in es, so, add many times query + # this is a temporary solution, the extract container will be improved in future. + def query_stats(job_id, times) + (1..times).each do |i| + new_job = ESQuery.new.query_by_id(job_id) + puts "query stats times: #{i}" + return new_job['stats'] if new_job['stats'] + + sleep 60 + end + + return nil + end + def record_jobs(job_id, job_commit) FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT commit_jobs = File.join(TMP_RESULT_ROOT, 'commit_jobs') -- 2.23.0

1 0

[PATCH lkp-tests 2/2] my_curl: change the curl request so that the cache can be used.
by Cui Yili 29 Dec '20

29 Dec '20

change curl reuqest, in order to use local cache. E..g https://download.redis.io/releases/redis-4.0.2.tar.gz ==> https://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k http://download.redis.io/releases/redis-4.0.2.tar.gz ==> http://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k Signed-off-by: cuiyili <2268260388(a)qq.com> --- rootfs/addon/usr/bin/my_curl | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 rootfs/addon/usr/bin/my_curl diff --git a/rootfs/addon/usr/bin/my_curl b/rootfs/addon/usr/bin/my_curl new file mode 100755 index 000000000..697f53d65 --- /dev/null +++ b/rootfs/addon/usr/bin/my_curl @@ -0,0 +1,22 @@ +#!/bin/bash + +declare -a array +j=0 + +if [ -n "$HTTP_PROXY_HOST" ]; then + for param in "$@"; + do + if [[ "$param" =~ http://(.*) ]] && [ -n "$HTTP_PROXY_PORT" ]; then + array[$j]="http://$HTTP_PROXY_HOST:$HTTP_PROXY_PORT/${BASH_REMATCH[1]}" + elif [[ "$param" =~ https://(.*) ]] && [ -n "$HTTPS_PROXY_PORT" ]; then + array[$j]="https://$HTTP_PROXY_HOST:$HTTPS_PROXY_PORT/${BASH_REMATCH[1]}" + else + array[$j]="$param" + fi + ((j++)) + done + + command curl -k ${array[*]} +else + command curl "$@" +fi -- 2.23.0

2 1

[PATCH lkp-tests 1/2] lkp-setup-rootfs: all use of env http_proxy and delete specific proxy settings
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- bin/lkp-setup-rootfs | 9 --------- 1 file changed, 9 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..a7f1a7ffb 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -105,7 +105,6 @@ install_proxy_yum() { [ -d "/etc/yum/" ] || return - echo "proxy=http://$SQUID_HOST:$SQUID_PORT" >> /etc/yum.conf if ls /etc/yum.repos.d/*.repo >/dev/null 2>&1; then sed -i 's/https:/http:/g' /etc/yum.repos.d/*.repo fi @@ -117,11 +116,6 @@ install_proxy_apt() { [ -d "/etc/apt/" ] || return - cat << EOF > /etc/apt/apt.conf.d/proxy -Acquire::http::proxy "http://$SQUID_HOST:$SQUID_PORT/"; -Acquire::ftp::proxy "ftp://$SQUID_HOST:$SQUID_PORT/"; -EOF - for cfg in /etc/apt/sources.list /etc/apt/sources.list.d/* do [ -f "$cfg" ] || continue @@ -176,9 +170,6 @@ install_proxy() install_goproxy - [ -z "$SQUID_HOST" ] && return - [ -z "$SQUID_PORT" ] && return - install_proxy_apt || install_proxy_yum } -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: don't return queues individually
by Ren Wen 29 Dec '20

29 Dec '20

- avoid to auto submit idle job if finding no job from sub-queues, because idle-queues are not found yet. - sub-queues should be shuffled. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 21396c4..a1e3497 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -72,14 +72,14 @@ class Sched end end - return sub_queues unless sub_queues.empty? - idle_queues = [] of String default_queues.each do |queue| idle_queues << "#{queue}/idle" end - return idle_queues + all_queues = rand_queues(sub_queues) + idle_queues + + return all_queues end def get_job_from_queues(queues, testbox) -- 2.23.0

1 0