January 2024 - Kernel - mailweb.openeuler.org

[PATCH v3 OLK-6.6 0/8] Introduce qos smt expeller for co-location
by Xia Fukun 03 Jan '24

03 Jan '24

We introduce the qos smt expeller, which lets online tasks to expel offline tasks on the smt sibling cpus, and exclusively occupy CPU resources.In this way we are able to improve QOS of online tasks in co-location. Guan Jing (8): sched: Introduce qos smt expeller for co-location sched: Implement the function of qos smt expeller sched: Add statistics for qos smt expeller sched: Add tracepoint for qos smt expeller config: Enable CONFIG_QOS_SCHED_SMT_EXPELLER sched/fair: Start tracking qos_offline tasks count in cfs_rq sched/fair: Introduce QOS_SMT_EXPELL priority reversion mechanism sched/fair: Add cmdline nosmtexpell arch/arm64/configs/openeuler_defconfig | 1 + arch/x86/configs/openeuler_defconfig | 1 + include/linux/sched.h | 12 + include/trace/events/sched.h | 55 ++++ init/Kconfig | 9 + kernel/sched/debug.c | 5 + kernel/sched/fair.c | 344 +++++++++++++++++++++++-- kernel/sched/sched.h | 27 ++ 8 files changed, 439 insertions(+), 15 deletions(-) -- 2.34.1

2 9

[PATCH OLK-6.6] drm/radeon: check the alloc_workqueue return value in radeon_crtc_init()
by Yang Yingliang 03 Jan '24

03 Jan '24

mainline inclusion from mainline-v6.8-rc1 commit 7a2464fac80d42f6f8819fed97a553e9c2f43310 category: bugfix bugzilla: https://gitee.com/src-openeuler/kernel/issues/I8T5EB CVE: CVE-2019-16230 Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… -------------------------------- check the alloc_workqueue return value in radeon_crtc_init() to avoid null-ptr-deref. Fixes: fa7f517cb26e ("drm/radeon: rework page flip handling v4") Signed-off-by: Yang Yingliang <yangyingliang(a)huawei.com> Signed-off-by: Alex Deucher <alexander.deucher(a)amd.com> --- drivers/gpu/drm/radeon/radeon_display.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/radeon/radeon_display.c b/drivers/gpu/drm/radeon/radeon_display.c index 901e75ec70ff..efd18c8d84c8 100644 --- a/drivers/gpu/drm/radeon/radeon_display.c +++ b/drivers/gpu/drm/radeon/radeon_display.c @@ -687,11 +687,16 @@ static void radeon_crtc_init(struct drm_device *dev, int index) if (radeon_crtc == NULL) return; + radeon_crtc->flip_queue = alloc_workqueue("radeon-crtc", WQ_HIGHPRI, 0); + if (!radeon_crtc->flip_queue) { + kfree(radeon_crtc); + return; + } + drm_crtc_init(dev, &radeon_crtc->base, &radeon_crtc_funcs); drm_mode_crtc_set_gamma_size(&radeon_crtc->base, 256); radeon_crtc->crtc_id = index; - radeon_crtc->flip_queue = alloc_workqueue("radeon-crtc", WQ_HIGHPRI, 0); rdev->mode_info.crtcs[index] = radeon_crtc; if (rdev->family >= CHIP_BONAIRE) { -- 2.25.1

2 1

[PATCH OLK-6.6 v5] tcp_comp: implement tcp compression
by Lu Wei 03 Jan '24

03 Jan '24

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I8T5DG CVE: NA ------------------------------------------------- Signed-off-by: Wang Yufen <wangyufen(a)huawei.com> Signed-off-by: Wei Yongjun <weiyongjun1(a)huawei.com> Signed-off-by: Lu Wei <luwei32(a)huawei.com> --- arch/arm64/configs/openeuler_defconfig | 1 + arch/x86/configs/openeuler_defconfig | 1 + include/linux/tcp.h | 8 +- include/net/inet_sock.h | 3 + include/net/sock.h | 1 + include/net/tcp.h | 40 ++ net/ipv4/Kconfig | 10 + net/ipv4/Makefile | 1 + net/ipv4/syncookies.c | 4 + net/ipv4/sysctl_net_ipv4.c | 42 ++ net/ipv4/tcp.c | 5 + net/ipv4/tcp_comp.c | 912 +++++++++++++++++++++++++ net/ipv4/tcp_input.c | 26 + net/ipv4/tcp_ipv4.c | 2 + net/ipv4/tcp_minisocks.c | 3 + net/ipv4/tcp_output.c | 52 ++ net/ipv6/syncookies.c | 4 + 17 files changed, 1114 insertions(+), 1 deletion(-) create mode 100644 net/ipv4/tcp_comp.c diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index 33ba39711884..abc33d2c29a6 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -1222,6 +1222,7 @@ CONFIG_DEFAULT_CUBIC=y # CONFIG_DEFAULT_RENO is not set CONFIG_DEFAULT_TCP_CONG="cubic" CONFIG_TCP_MD5SIG=y +CONFIG_TCP_COMP=y CONFIG_IPV6=y CONFIG_IPV6_ROUTER_PREF=y CONFIG_IPV6_ROUTE_INFO=y diff --git a/arch/x86/configs/openeuler_defconfig b/arch/x86/configs/openeuler_defconfig index 44040b835333..177a6fdcce58 100644 --- a/arch/x86/configs/openeuler_defconfig +++ b/arch/x86/configs/openeuler_defconfig @@ -1244,6 +1244,7 @@ CONFIG_DEFAULT_CUBIC=y # CONFIG_DEFAULT_RENO is not set CONFIG_DEFAULT_TCP_CONG="cubic" CONFIG_TCP_MD5SIG=y +CONFIG_TCP_COMP=y CONFIG_IPV6=y CONFIG_IPV6_ROUTER_PREF=y CONFIG_IPV6_ROUTE_INFO=y diff --git a/include/linux/tcp.h b/include/linux/tcp.h index 3c5efeeb024f..d94a84c291f0 100644 --- a/include/linux/tcp.h +++ b/include/linux/tcp.h @@ -123,7 +123,10 @@ struct tcp_options_received { snd_wscale : 4, /* Window scaling received from sender */ rcv_wscale : 4; /* Window scaling to send to receiver */ u8 saw_unknown:1, /* Received unknown option */ - unused:7; +#if IS_ENABLED(CONFIG_TCP_COMP) + comp_ok:1, /* COMP seen on SYN packet */ +#endif + unused:6; u8 num_sacks; /* Number of SACK blocks */ u16 user_mss; /* mss requested by user in ioctl */ u16 mss_clamp; /* Maximal mss, negotiated at connection setup */ @@ -136,6 +139,9 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt) #if IS_ENABLED(CONFIG_SMC) rx_opt->smc_ok = 0; #endif +#if IS_ENABLED(CONFIG_TCP_COMP) + rx_opt->comp_ok = 0; +#endif } /* This is the max number of SACKS that we'll generate and process. It's safe diff --git a/include/net/inet_sock.h b/include/net/inet_sock.h index 2de0e4d4a027..13c1f1e83de5 100644 --- a/include/net/inet_sock.h +++ b/include/net/inet_sock.h @@ -87,6 +87,9 @@ struct inet_request_sock { ecn_ok : 1, acked : 1, no_srccheck: 1, +#if IS_ENABLED(CONFIG_TCP_COMP) + comp_ok : 1, +#endif smc_ok : 1; u32 ir_mark; union { diff --git a/include/net/sock.h b/include/net/sock.h index 7753354d59c0..c86845136ec5 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -959,6 +959,7 @@ enum sock_flags { SOCK_XDP, /* XDP is attached */ SOCK_TSTAMP_NEW, /* Indicates 64 bit timestamps always */ SOCK_RCVMARK, /* Receive SO_MARK ancillary data with packet */ + SOCK_COMP, }; #define SK_FLAGS_TIMESTAMP ((1UL << SOCK_TIMESTAMP) | (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE)) diff --git a/include/net/tcp.h b/include/net/tcp.h index e9d387fffe22..cb33a2c46b2f 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -195,6 +195,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo); */ #define TCPOPT_FASTOPEN_MAGIC 0xF989 #define TCPOPT_SMC_MAGIC 0xE2D4C3D9 +#define TCPOPT_COMP_MAGIC 0x7954 /* * TCP option lengths @@ -208,6 +209,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo); #define TCPOLEN_FASTOPEN_BASE 2 #define TCPOLEN_EXP_FASTOPEN_BASE 4 #define TCPOLEN_EXP_SMC_BASE 6 +#define TCPOLEN_EXP_COMP_BASE 4 /* But this is what stacks really send out. */ #define TCPOLEN_TSTAMP_ALIGNED 12 @@ -2531,4 +2533,42 @@ static inline u64 tcp_transmit_time(const struct sock *sk) return 0; } +#if IS_ENABLED(CONFIG_TCP_COMP) +extern struct static_key_false tcp_have_comp; + +extern unsigned long *sysctl_tcp_compression_ports; +extern int sysctl_tcp_compression_local; + +bool tcp_syn_comp_enabled(struct sock *sk); +bool tcp_synack_comp_enabled(struct sock *sk, + const struct inet_request_sock *ireq); +void tcp_init_compression(struct sock *sk); +void tcp_cleanup_compression(struct sock *sk); +int tcp_comp_init(void); +#else +static inline bool tcp_syn_comp_enabled(struct tcp_sock *tp) +{ + return false; +} + +static inline bool tcp_synack_comp_enabled(struct sock *sk, + const struct inet_request_sock *ireq) +{ + return false; +} + +static inline void tcp_init_compression(struct sock *sk) +{ +} + +static inline void tcp_cleanup_compression(struct sock *sk) +{ +} + +static inline int tcp_comp_init(void) +{ + return 0; +} +#endif + #endif /* _TCP_H */ diff --git a/net/ipv4/Kconfig b/net/ipv4/Kconfig index 2dfb12230f08..a4405ea38338 100644 --- a/net/ipv4/Kconfig +++ b/net/ipv4/Kconfig @@ -751,3 +751,13 @@ config TCP_MD5SIG on the Internet. If unsure, say N. + +config TCP_COMP + bool "TCP: Transport Layer Compression support" + depends on CRYPTO_ZSTD=y + select STREAM_PARSER + help + Enable kernel payload compression support for TCP protocol. This allows + payload compression handling of the TCP protocol to be done in-kernel. + + If unsure, say Y. diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile index b18ba8ef93ad..aa458d9f534a 100644 --- a/net/ipv4/Makefile +++ b/net/ipv4/Makefile @@ -65,6 +65,7 @@ obj-$(CONFIG_TCP_CONG_ILLINOIS) += tcp_illinois.o obj-$(CONFIG_NET_SOCK_MSG) += tcp_bpf.o obj-$(CONFIG_BPF_SYSCALL) += udp_bpf.o obj-$(CONFIG_NETLABEL) += cipso_ipv4.o +obj-$(CONFIG_TCP_COMP) += tcp_comp.o obj-$(CONFIG_XFRM) += xfrm4_policy.o xfrm4_state.o xfrm4_input.o \ xfrm4_output.o xfrm4_protocol.o diff --git a/net/ipv4/syncookies.c b/net/ipv4/syncookies.c index 3b4dafefb4b0..584b29c154e8 100644 --- a/net/ipv4/syncookies.c +++ b/net/ipv4/syncookies.c @@ -391,6 +391,10 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb) if (IS_ENABLED(CONFIG_SMC)) ireq->smc_ok = 0; +#if IS_ENABLED(CONFIG_TCP_COMP) + ireq->comp_ok = 0; +#endif + ireq->ir_iif = inet_request_bound_dev_if(sk, skb); /* We throwed the options of the initial SYN away, so we hope diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c index b17eb28a9690..f212133b05e9 100644 --- a/net/ipv4/sysctl_net_ipv4.c +++ b/net/ipv4/sysctl_net_ipv4.c @@ -469,6 +469,30 @@ static int proc_fib_multipath_hash_fields(struct ctl_table *table, int write, } #endif +#if IS_ENABLED(CONFIG_TCP_COMP) +static int proc_tcp_compression_ports(struct ctl_table *table, int write, + void __user *buffer, size_t *lenp, + loff_t *ppos) +{ + unsigned long *bitmap = *(unsigned long **)table->data; + unsigned long bitmap_len = table->maxlen; + int ret; + + ret = proc_do_large_bitmap(table, write, buffer, lenp, ppos); + if (write && ret == 0) { + if (bitmap_empty(bitmap, bitmap_len)) { + if (static_key_enabled(&tcp_have_comp)) + static_branch_disable(&tcp_have_comp); + } else { + if (!static_key_enabled(&tcp_have_comp)) + static_branch_enable(&tcp_have_comp); + } + } + + return ret; +} +#endif + static struct ctl_table ipv4_table[] = { { .procname = "tcp_max_orphans", @@ -587,6 +611,24 @@ static struct ctl_table ipv4_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, +#if IS_ENABLED(CONFIG_TCP_COMP) + { + .procname = "tcp_compression_ports", + .data = &sysctl_tcp_compression_ports, + .maxlen = 65536, + .mode = 0644, + .proc_handler = proc_tcp_compression_ports, + }, + { + .procname = "tcp_compression_local", + .data = &sysctl_tcp_compression_local, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, +#endif { } }; diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 3d3a24f79573..2703be8a7316 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -302,6 +302,10 @@ DEFINE_STATIC_KEY_FALSE(tcp_have_smc); EXPORT_SYMBOL(tcp_have_smc); #endif +#if IS_ENABLED(CONFIG_TCP_COMP) +DEFINE_STATIC_KEY_FALSE(tcp_have_comp); +#endif + /* * Current number of TCP sockets. */ @@ -4707,5 +4711,6 @@ void __init tcp_init(void) tcp_metrics_init(); BUG_ON(tcp_register_congestion_control(&tcp_reno) != 0); tcp_tasklet_init(); + tcp_comp_init(); mptcp_init(); } diff --git a/net/ipv4/tcp_comp.c b/net/ipv4/tcp_comp.c new file mode 100644 index 000000000000..53157a413b58 --- /dev/null +++ b/net/ipv4/tcp_comp.c @@ -0,0 +1,912 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * TCP compression support + * + * Copyright(c) 2021 Huawei Technologies Co., Ltd + */ + +#include <linux/skmsg.h> +#include <linux/zstd.h> + +#define TCP_COMP_MAX_PADDING 64 +#define TCP_COMP_DATA_SIZE 65536 +#define TCP_COMP_SCRATCH_SIZE (TCP_COMP_DATA_SIZE - 1) +#define TCP_COMP_MAX_CSIZE (TCP_COMP_SCRATCH_SIZE + TCP_COMP_MAX_PADDING) +#define TCP_COMP_ALLOC_ORDER get_order(TCP_COMP_DATA_SIZE) +#define TCP_COMP_MAX_WINDOWLOG 17 +#define TCP_COMP_MAX_INPUT (1 << TCP_COMP_MAX_WINDOWLOG) + +#define TCP_COMP_SEND_PENDING 1 +#define ZSTD_COMP_DEFAULT_LEVEL 1 + +static unsigned long tcp_compression_ports[65536 / 8]; + +unsigned long *sysctl_tcp_compression_ports = tcp_compression_ports; +int sysctl_tcp_compression_local __read_mostly; + +static struct proto tcp_prot_override; + +struct tcp_comp_context_tx { + ZSTD_CStream *cstream; + void *cworkspace; + void *plaintext_data; + void *compressed_data; + struct sk_msg msg; + bool in_tcp_sendpages; +}; + +struct tcp_comp_context_rx { + ZSTD_DStream *dstream; + void *dworkspace; + void *plaintext_data; + + struct strparser strp; + void (*saved_data_ready)(struct sock *sk); + struct sk_buff *pkt; + struct sk_buff *dpkt; +}; + +struct tcp_comp_context { + struct rcu_head rcu; + + struct proto *sk_proto; + void (*sk_write_space)(struct sock *sk); + + struct tcp_comp_context_tx tx; + struct tcp_comp_context_rx rx; + + unsigned long flags; +}; + +static bool tcp_comp_is_write_pending(struct tcp_comp_context *ctx) +{ + return test_bit(TCP_COMP_SEND_PENDING, &ctx->flags); +} + +static void tcp_comp_err_abort(struct sock *sk, int err) +{ + sk->sk_err = err; + sk->sk_error_report(sk); +} + +static bool tcp_comp_enabled(__be32 saddr, __be32 daddr, int port) +{ + if (!sysctl_tcp_compression_local && + (saddr == daddr || ipv4_is_loopback(daddr))) + return false; + + return test_bit(port, sysctl_tcp_compression_ports); +} + +bool tcp_syn_comp_enabled(struct sock *sk) +{ + struct inet_sock *inet = inet_sk(sk); + + return tcp_comp_enabled(inet->inet_saddr, inet->inet_daddr, + ntohs(inet->inet_dport)); +} + +bool tcp_synack_comp_enabled(struct sock *sk, + const struct inet_request_sock *ireq) +{ + struct inet_sock *inet = inet_sk(sk); + + if (!ireq->comp_ok) + return false; + + return tcp_comp_enabled(ireq->ir_loc_addr, ireq->ir_rmt_addr, + ntohs(inet->inet_sport)); +} + +static struct tcp_comp_context *comp_get_ctx(const struct sock *sk) +{ + struct inet_connection_sock *icsk = inet_csk(sk); + + return (__force void *)icsk->icsk_ulp_data; +} + +static int tcp_comp_tx_context_init(struct tcp_comp_context *ctx) +{ + ZSTD_parameters params; + int csize; + + params = ZSTD_getParams(ZSTD_COMP_DEFAULT_LEVEL, PAGE_SIZE, 0); + csize = zstd_cstream_workspace_bound(&params.cParams); + if (csize <= 0) + return -EINVAL; + + ctx->tx.cworkspace = kmalloc(csize, GFP_KERNEL); + if (!ctx->tx.cworkspace) + return -ENOMEM; + + ctx->tx.cstream = zstd_init_cstream(&params, 0, ctx->tx.cworkspace, + csize); + if (!ctx->tx.cstream) + goto err_cstream; + + ctx->tx.plaintext_data = kvmalloc(TCP_COMP_SCRATCH_SIZE, GFP_KERNEL); + if (!ctx->tx.plaintext_data) + goto err_cstream; + + ctx->tx.compressed_data = kvmalloc(TCP_COMP_MAX_CSIZE, GFP_KERNEL); + if (!ctx->tx.compressed_data) + goto err_compressed; + + return 0; + +err_compressed: + kvfree(ctx->tx.plaintext_data); + ctx->tx.plaintext_data = NULL; +err_cstream: + kfree(ctx->tx.cworkspace); + ctx->tx.cworkspace = NULL; + + return -ENOMEM; +} + +static void *tcp_comp_get_tx_stream(struct sock *sk) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + + if (!ctx->tx.plaintext_data) + tcp_comp_tx_context_init(ctx); + + return ctx->tx.plaintext_data; +} + +static int alloc_compressed_msg(struct sock *sk, int len) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct sk_msg *msg = &ctx->tx.msg; + + sk_msg_init(msg); + + return sk_msg_alloc(sk, msg, len, 0); +} + +static int memcopy_from_iter(struct sock *sk, struct iov_iter *from, int copy) +{ + void *dest; + int rc; + + dest = tcp_comp_get_tx_stream(sk); + if (!dest) + return -ENOSPC; + + if (sk->sk_route_caps & NETIF_F_NOCACHE_COPY) + rc = copy_from_iter_nocache(dest, copy, from); + else + rc = copy_from_iter(dest, copy, from); + + if (rc != copy) + rc = -EFAULT; + + return rc; +} + +static int memcopy_to_msg(struct sock *sk, int bytes) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct sk_msg *msg = &ctx->tx.msg; + int i = msg->sg.curr; + struct scatterlist *sge; + u32 copy, buf_size; + void *from, *to; + + from = ctx->tx.compressed_data; + do { + sge = sk_msg_elem(msg, i); + /* This is possible if a trim operation shrunk the buffer */ + if (msg->sg.copybreak >= sge->length) { + msg->sg.copybreak = 0; + sk_msg_iter_var_next(i); + if (i == msg->sg.end) + break; + sge = sk_msg_elem(msg, i); + } + buf_size = sge->length - msg->sg.copybreak; + copy = (buf_size > bytes) ? bytes : buf_size; + to = sg_virt(sge) + msg->sg.copybreak; + msg->sg.copybreak += copy; + memcpy(to, from, copy); + bytes -= copy; + from += copy; + if (!bytes) + break; + msg->sg.copybreak = 0; + sk_msg_iter_var_next(i); + } while (i != msg->sg.end); + + msg->sg.curr = i; + return bytes; +} + +static int tcp_comp_compress_to_msg(struct sock *sk, int bytes) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + ZSTD_outBuffer outbuf; + ZSTD_inBuffer inbuf; + size_t ret; + + inbuf.src = ctx->tx.plaintext_data; + outbuf.dst = ctx->tx.compressed_data; + inbuf.size = bytes; + outbuf.size = TCP_COMP_MAX_CSIZE; + inbuf.pos = 0; + outbuf.pos = 0; + + ret = ZSTD_compressStream(ctx->tx.cstream, &outbuf, &inbuf); + if (ZSTD_isError(ret)) + return -EIO; + + ret = ZSTD_flushStream(ctx->tx.cstream, &outbuf); + if (ZSTD_isError(ret)) + return -EIO; + + if (inbuf.pos != inbuf.size) + return -EIO; + + if (memcopy_to_msg(sk, outbuf.pos)) + return -EIO; + + sk_msg_trim(sk, &ctx->tx.msg, outbuf.pos); + + return 0; +} + +static int tcp_comp_push_msg(struct sock *sk, struct sk_msg *msg, int flags) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct msghdr mh; + struct bio_vec bvec; + struct scatterlist *sg; + int ret, offset; + struct page *p; + size_t size; + + ctx->tx.in_tcp_sendpages = true; + while (1) { + sg = sk_msg_elem(msg, msg->sg.start); + offset = sg->offset; + size = sg->length; + p = sg_page(sg); +retry: + memset(&mh, 0, sizeof(struct msghdr)); + memset(&bvec, 0, sizeof(struct bio_vec)); + + mh.msg_flags = flags | MSG_SPLICE_PAGES; + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&mh.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &mh, size); + if (ret != size) { + if (ret > 0) { + sk_mem_uncharge(sk, ret); + sg->offset += ret; + sg->length -= ret; + size -= ret; + offset += ret; + goto retry; + } + ctx->tx.in_tcp_sendpages = false; + return ret; + } + + sk_mem_uncharge(sk, ret); + msg->sg.size -= size; + put_page(p); + sk_msg_iter_next(msg, start); + if (msg->sg.start == msg->sg.end) + break; + } + + clear_bit(TCP_COMP_SEND_PENDING, &ctx->flags); + ctx->tx.in_tcp_sendpages = false; + + return 0; +} + +static int tcp_comp_push(struct sock *sk, int bytes, int flags) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + int ret; + + ret = tcp_comp_compress_to_msg(sk, bytes); + if (ret < 0) { + pr_debug("%s: failed to compress sg\n", __func__); + return ret; + } + + set_bit(TCP_COMP_SEND_PENDING, &ctx->flags); + + ret = tcp_comp_push_msg(sk, &ctx->tx.msg, flags); + if (ret) { + pr_debug("%s: failed to tcp_comp_push_sg\n", __func__); + return ret; + } + + return 0; +} + +static int wait_on_pending_writer(struct sock *sk, long *timeo) +{ + DEFINE_WAIT_FUNC(wait, woken_wake_function); + int ret = 0; + + add_wait_queue(sk_sleep(sk), &wait); + while (1) { + if (!*timeo) { + ret = -EAGAIN; + break; + } + + if (signal_pending(current)) { + ret = sock_intr_errno(*timeo); + break; + } + + if (sk_wait_event(sk, timeo, !sk->sk_write_pending, &wait)) + break; + } + remove_wait_queue(sk_sleep(sk), &wait); + + return ret; +} + +static int tcp_comp_push_pending_msg(struct sock *sk, int flags) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct sk_msg *msg = &ctx->tx.msg; + + if (msg->sg.start == msg->sg.end) + return 0; + + return tcp_comp_push_msg(sk, msg, flags); +} + +static int tcp_comp_complete_pending_work(struct sock *sk, int flags, + long *timeo) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + int ret = 0; + + if (unlikely(sk->sk_write_pending)) + ret = wait_on_pending_writer(sk, timeo); + + if (!ret && tcp_comp_is_write_pending(ctx)) + ret = tcp_comp_push_pending_msg(sk, flags); + + return ret; +} + +static int tcp_comp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + int copied = 0, err = 0; + size_t try_to_copy; + int required_size; + long timeo; + + lock_sock(sk); + + timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); + + err = tcp_comp_complete_pending_work(sk, msg->msg_flags, &timeo); + if (err) + goto out_err; + + while (msg_data_left(msg)) { + if (sk->sk_err) { + err = -sk->sk_err; + goto out_err; + } + + try_to_copy = msg_data_left(msg); + if (try_to_copy > TCP_COMP_SCRATCH_SIZE) + try_to_copy = TCP_COMP_SCRATCH_SIZE; + required_size = try_to_copy + TCP_COMP_MAX_PADDING; + + if (!sk_stream_memory_free(sk)) + goto wait_for_sndbuf; + +alloc_compressed: + err = alloc_compressed_msg(sk, required_size); + if (err) { + if (err != -ENOSPC) + goto wait_for_memory; + goto out_err; + } + + err = memcopy_from_iter(sk, &msg->msg_iter, try_to_copy); + if (err < 0) + goto out_err; + + copied += try_to_copy; + + err = tcp_comp_push(sk, try_to_copy, msg->msg_flags); + if (err < 0) { + if (err == -ENOMEM) + goto wait_for_memory; + if (err != -EAGAIN) + tcp_comp_err_abort(sk, EBADMSG); + goto out_err; + } + + continue; +wait_for_sndbuf: + set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); +wait_for_memory: + err = sk_stream_wait_memory(sk, &timeo); + if (err) + goto out_err; + if (ctx->tx.msg.sg.size < required_size) + goto alloc_compressed; + } + +out_err: + err = sk_stream_error(sk, msg->msg_flags, err); + + release_sock(sk); + + return copied ? copied : err; +} + +static struct sk_buff *comp_wait_data(struct sock *sk, int flags, + long timeo, int *err) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct sk_buff *skb; + DEFINE_WAIT_FUNC(wait, woken_wake_function); + + while (!(skb = ctx->rx.pkt)) { + if (sk->sk_err) { + *err = sock_error(sk); + return NULL; + } + + if (!skb_queue_empty(&sk->sk_receive_queue)) { + __strp_unpause(&ctx->rx.strp); + if (ctx->rx.pkt) + return ctx->rx.pkt; + } + + if (sk->sk_shutdown & RCV_SHUTDOWN) + return NULL; + + if (sock_flag(sk, SOCK_DONE)) + return NULL; + + if ((flags & MSG_DONTWAIT) || !timeo) { + *err = -EAGAIN; + return NULL; + } + + add_wait_queue(sk_sleep(sk), &wait); + sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk); + sk_wait_event(sk, &timeo, ctx->rx.pkt != skb, &wait); + sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk); + remove_wait_queue(sk_sleep(sk), &wait); + + /* Handle signals */ + if (signal_pending(current)) { + *err = sock_intr_errno(timeo); + return NULL; + } + } + + return skb; +} + +static bool comp_advance_skb(struct sock *sk, struct sk_buff *skb, + unsigned int len) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct strp_msg *rxm = strp_msg(skb); + + if (len < rxm->full_len) { + rxm->offset += len; + rxm->full_len -= len; + return false; + } + + /* Finished with message */ + ctx->rx.pkt = NULL; + kfree_skb(skb); + __strp_unpause(&ctx->rx.strp); + + return true; +} + +static bool comp_advance_dskb(struct sock *sk, struct sk_buff *skb, + unsigned int len) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct strp_msg *rxm = strp_msg(skb); + + if (len < rxm->full_len) { + rxm->offset += len; + rxm->full_len -= len; + return false; + } + + /* Finished with message */ + ctx->rx.dpkt = NULL; + kfree_skb(skb); + return true; +} + +static int tcp_comp_rx_context_init(struct tcp_comp_context *ctx) +{ + int dsize; + + dsize = zstd_dstream_workspace_bound(TCP_COMP_MAX_INPUT); + if (dsize <= 0) + return -EINVAL; + + ctx->rx.dworkspace = kmalloc(dsize, GFP_KERNEL); + if (!ctx->rx.dworkspace) + return -ENOMEM; + + ctx->rx.dstream = zstd_init_dstream(TCP_COMP_MAX_INPUT, + ctx->rx.dworkspace, dsize); + if (!ctx->rx.dstream) + goto err_dstream; + + ctx->rx.plaintext_data = kvmalloc(TCP_COMP_MAX_CSIZE * 32, GFP_KERNEL); + if (!ctx->rx.plaintext_data) + goto err_dstream; + + return 0; + +err_dstream: + kfree(ctx->rx.dworkspace); + ctx->rx.dworkspace = NULL; + + return -ENOMEM; +} + +static void *tcp_comp_get_rx_stream(struct sock *sk) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + + if (!ctx->rx.plaintext_data) + tcp_comp_rx_context_init(ctx); + + return ctx->rx.plaintext_data; +} + +static int tcp_comp_decompress(struct sock *sk, struct sk_buff *skb, int flags) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct strp_msg *rxm = strp_msg(skb); + size_t ret, compressed_len = 0; + int nr_frags_over = 0; + ZSTD_outBuffer outbuf; + ZSTD_inBuffer inbuf; + struct sk_buff *nskb; + int len, plen; + void *to; + + to = tcp_comp_get_rx_stream(sk); + if (!to) + return -ENOSPC; + + if (skb_linearize_cow(skb)) + return -ENOMEM; + + nskb = skb_copy(skb, GFP_KERNEL); + if (!nskb) + return -ENOMEM; + + while (compressed_len < (skb->len - rxm->offset)) { + if (skb_shinfo(nskb)->nr_frags >= MAX_SKB_FRAGS) + break; + + len = 0; + plen = skb->len - rxm->offset - compressed_len; + if (plen > TCP_COMP_MAX_CSIZE) + plen = TCP_COMP_MAX_CSIZE; + + inbuf.src = (char *)skb->data + rxm->offset + compressed_len; + inbuf.pos = 0; + inbuf.size = plen; + + outbuf.dst = ctx->rx.plaintext_data; + outbuf.pos = 0; + outbuf.size = MAX_SKB_FRAGS * TCP_COMP_DATA_SIZE; + outbuf.size -= skb_shinfo(nskb)->nr_frags * TCP_COMP_DATA_SIZE; + + ret = ZSTD_decompressStream(ctx->rx.dstream, &outbuf, &inbuf); + if (ZSTD_isError(ret)) { + kfree_skb(nskb); + return -EIO; + } + + if (!compressed_len) { + len = outbuf.pos - skb->len; + if (len > skb_tailroom(nskb)) + len = skb_tailroom(nskb); + + __skb_put(nskb, len); + + len += skb->len; + skb_copy_to_linear_data(nskb, to, len); + } + + while ((to += len, outbuf.pos -= len) > 0) { + struct page *pages; + skb_frag_t *frag; + + if (skb_shinfo(nskb)->nr_frags >= MAX_SKB_FRAGS) { + nr_frags_over = 1; + break; + } + + frag = skb_shinfo(nskb)->frags + + skb_shinfo(nskb)->nr_frags; + pages = alloc_pages(__GFP_NOWARN | GFP_KERNEL | __GFP_COMP, + TCP_COMP_ALLOC_ORDER); + if (!pages) { + kfree_skb(nskb); + return -ENOMEM; + } + + frag->bv_page = pages; + len = PAGE_SIZE << TCP_COMP_ALLOC_ORDER; + if (outbuf.pos < len) + len = outbuf.pos; + + frag->bv_offset = 0; + skb_frag_size_set(frag, len); + memcpy(skb_frag_address(frag), to, len); + + nskb->truesize += len; + nskb->data_len += len; + nskb->len += len; + skb_shinfo(nskb)->nr_frags++; + } + + if (nr_frags_over) + break; + + compressed_len += inbuf.pos; + } + + ctx->rx.dpkt = nskb; + rxm = strp_msg(nskb); + rxm->full_len = nskb->len; + rxm->offset = 0; + comp_advance_skb(sk, skb, compressed_len); + + return 0; +} + +static int tcp_comp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, + int flags, int *addr_len) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + struct strp_msg *rxm; + struct sk_buff *skb; + ssize_t copied = 0; + int target, err = 0; + long timeo; + + if (unlikely(flags & MSG_ERRQUEUE)) + return sock_recv_errqueue(sk, msg, len, SOL_IP, IP_RECVERR); + + lock_sock(sk); + + target = sock_rcvlowat(sk, flags & MSG_WAITALL, len); + timeo = sock_rcvtimeo(sk, flags & MSG_WAITALL); + + do { + int chunk = 0; + + if (!ctx->rx.dpkt) { + skb = comp_wait_data(sk, flags, timeo, &err); + if (!skb) + goto recv_end; + + err = tcp_comp_decompress(sk, skb, flags); + if (err < 0) { + goto recv_end; + } + } + + skb = ctx->rx.dpkt; + rxm = strp_msg(skb); + chunk = min_t(unsigned int, rxm->full_len, len); + err = skb_copy_datagram_msg(skb, rxm->offset, msg, + chunk); + if (err < 0) + goto recv_end; + + copied += chunk; + len -= chunk; + if (likely(!(flags & MSG_PEEK))) + comp_advance_dskb(sk, skb, chunk); + else + break; + + if (copied >= target && !ctx->rx.dpkt) + break; + } while (len > 0); + +recv_end: + release_sock(sk); + return copied ? : err; +} + +bool comp_stream_read(struct sock *sk) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + + if (!ctx) + return false; + + if (ctx->rx.pkt || ctx->rx.dpkt) + return true; + + return false; +} + +static void comp_data_ready(struct sock *sk) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + + strp_data_ready(&ctx->rx.strp); +} + +static void comp_queue(struct strparser *strp, struct sk_buff *skb) +{ + struct tcp_comp_context *ctx = comp_get_ctx(strp->sk); + + ctx->rx.pkt = skb; + strp_pause(strp); + ctx->rx.saved_data_ready(strp->sk); +} + +static int comp_read_size(struct strparser *strp, struct sk_buff *skb) +{ + struct strp_msg *rxm = strp_msg(skb); + + if (rxm->offset > skb->len) + return 0; + + return skb->len - rxm->offset; +} + +void comp_setup_strp(struct sock *sk, struct tcp_comp_context *ctx) +{ + struct strp_callbacks cb; + + memset(&cb, 0, sizeof(cb)); + cb.rcv_msg = comp_queue; + cb.parse_msg = comp_read_size; + strp_init(&ctx->rx.strp, sk, &cb); + + write_lock_bh(&sk->sk_callback_lock); + ctx->rx.saved_data_ready = sk->sk_data_ready; + sk->sk_data_ready = comp_data_ready; + write_unlock_bh(&sk->sk_callback_lock); + + strp_check_rcv(&ctx->rx.strp); +} + +static void tcp_comp_write_space(struct sock *sk) +{ + struct tcp_comp_context *ctx = comp_get_ctx(sk); + + if (ctx->tx.in_tcp_sendpages) { + ctx->sk_write_space(sk); + return; + } + + if (!sk->sk_write_pending && tcp_comp_is_write_pending(ctx)) { + gfp_t sk_allocation = sk->sk_allocation; + int rc; + + sk->sk_allocation = GFP_ATOMIC; + rc = tcp_comp_push_pending_msg(sk, MSG_DONTWAIT | MSG_NOSIGNAL); + sk->sk_allocation = sk_allocation; + + if (rc < 0) + return; + } + + ctx->sk_write_space(sk); +} + +void tcp_init_compression(struct sock *sk) +{ + struct inet_connection_sock *icsk = inet_csk(sk); + struct tcp_comp_context *ctx = NULL; + struct sk_msg *msg = NULL; + struct tcp_sock *tp = tcp_sk(sk); + + if (!tp->rx_opt.comp_ok) + return; + + ctx = kzalloc(sizeof(*ctx), GFP_ATOMIC); + if (!ctx) + return; + + msg = &ctx->tx.msg; + sk_msg_init(msg); + + ctx->sk_write_space = sk->sk_write_space; + ctx->sk_proto = sk->sk_prot; + WRITE_ONCE(sk->sk_prot, &tcp_prot_override); + sk->sk_write_space = tcp_comp_write_space; + + rcu_assign_pointer(icsk->icsk_ulp_data, ctx); + + sock_set_flag(sk, SOCK_COMP); + comp_setup_strp(sk, ctx); +} + +static void tcp_comp_context_tx_free(struct tcp_comp_context *ctx) +{ + kfree(ctx->tx.cworkspace); + ctx->tx.cworkspace = NULL; + + kvfree(ctx->tx.plaintext_data); + ctx->tx.plaintext_data = NULL; + + kvfree(ctx->tx.compressed_data); + ctx->tx.compressed_data = NULL; +} + +static void tcp_comp_context_rx_free(struct tcp_comp_context *ctx) +{ + kfree(ctx->rx.dworkspace); + ctx->rx.dworkspace = NULL; + + kvfree(ctx->rx.plaintext_data); + ctx->rx.plaintext_data = NULL; +} + +static void tcp_comp_context_free(struct rcu_head *head) +{ + struct tcp_comp_context *ctx; + + ctx = container_of(head, struct tcp_comp_context, rcu); + + tcp_comp_context_tx_free(ctx); + tcp_comp_context_rx_free(ctx); + strp_done(&ctx->rx.strp); + kfree(ctx); +} + +void tcp_cleanup_compression(struct sock *sk) +{ + struct inet_connection_sock *icsk = inet_csk(sk); + struct tcp_comp_context *ctx = comp_get_ctx(sk); + + if (!ctx || !sock_flag(sk, SOCK_COMP)) + return; + + if (ctx->rx.pkt) { + kfree_skb(ctx->rx.pkt); + ctx->rx.pkt = NULL; + } + + if (ctx->rx.dpkt) { + kfree_skb(ctx->rx.dpkt); + ctx->rx.dpkt = NULL; + } + strp_stop(&ctx->rx.strp); + + rcu_assign_pointer(icsk->icsk_ulp_data, NULL); + call_rcu(&ctx->rcu, tcp_comp_context_free); +} + +int tcp_comp_init(void) +{ + tcp_prot_override = tcp_prot; + tcp_prot_override.sendmsg = tcp_comp_sendmsg; + tcp_prot_override.recvmsg = tcp_comp_recvmsg; + tcp_prot_override.sock_is_readable = comp_stream_read; + + return 0; +} diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index fd5c13c1fbc8..a5adf4822663 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -4012,6 +4012,24 @@ static bool smc_parse_options(const struct tcphdr *th, return false; } +static bool tcp_parse_comp_option(const struct tcphdr *th, + struct tcp_options_received *opt_rx, + const unsigned char *ptr, + int opsize) +{ +#if IS_ENABLED(CONFIG_TCP_COMP) + if (static_branch_unlikely(&tcp_have_comp)) { + if (th->syn && !(opsize & 1) && + opsize >= TCPOLEN_EXP_COMP_BASE && + get_unaligned_be16(ptr) == TCPOPT_COMP_MAGIC) { + opt_rx->comp_ok = 1; + return true; + } + } +#endif + return false; +} + /* Try to parse the MSS option from the TCP header. Return 0 on failure, clamped * value on success. */ @@ -4171,6 +4189,10 @@ void tcp_parse_options(const struct net *net, if (smc_parse_options(th, opt_rx, ptr, opsize)) break; + if (tcp_parse_comp_option(th, opt_rx, ptr, + opsize)) + break; + opt_rx->saw_unknown = 1; break; @@ -6097,6 +6119,7 @@ void tcp_init_transfer(struct sock *sk, int bpf_op, struct sk_buff *skb) /* Initialize congestion control unless BPF initialized it already: */ if (!icsk->icsk_ca_initialized) tcp_init_congestion_control(sk); + tcp_init_compression(sk); tcp_init_buffer_space(sk); } @@ -6835,6 +6858,9 @@ static void tcp_openreq_init(struct request_sock *req, ireq->smc_ok = rx_opt->smc_ok && !(tcp_sk(sk)->smc_hs_congested && tcp_sk(sk)->smc_hs_congested(sk)); #endif +#if IS_ENABLED(CONFIG_TCP_COMP) + ireq->comp_ok = rx_opt->comp_ok; +#endif } struct request_sock *inet_reqsk_alloc(const struct request_sock_ops *ops, diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 4167e8a48b60..9cbf23c1c039 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -2304,6 +2304,8 @@ void tcp_v4_destroy_sock(struct sock *sk) tcp_cleanup_congestion_control(sk); + tcp_cleanup_compression(sk); + tcp_cleanup_ulp(sk); /* Cleanup up the write buffer. */ diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c index b98d476f1594..d67a84c114ce 100644 --- a/net/ipv4/tcp_minisocks.c +++ b/net/ipv4/tcp_minisocks.c @@ -543,6 +543,9 @@ struct sock *tcp_create_openreq_child(const struct sock *sk, newtp->rcv_ssthresh = req->rsk_rcv_wnd; newtp->rcv_wnd = req->rsk_rcv_wnd; newtp->rx_opt.wscale_ok = ireq->wscale_ok; +#if IS_ENABLED(CONFIG_TCP_COMP) + newtp->rx_opt.comp_ok = ireq->comp_ok; +#endif if (newtp->rx_opt.wscale_ok) { newtp->rx_opt.snd_wscale = ireq->snd_wscale; newtp->rx_opt.rcv_wscale = ireq->rcv_wscale; diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 1917c62ad3bf..eee34f6c7643 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -422,6 +422,7 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp) #define OPTION_FAST_OPEN_COOKIE BIT(8) #define OPTION_SMC BIT(9) #define OPTION_MPTCP BIT(10) +#define OPTION_COMP BIT(11) static void smc_options_write(__be32 *ptr, u16 *options) { @@ -438,6 +439,19 @@ static void smc_options_write(__be32 *ptr, u16 *options) #endif } +static void comp_options_write(__be32 *ptr, u16 *options) +{ +#if IS_ENABLED(CONFIG_TCP_COMP) + if (static_branch_unlikely(&tcp_have_comp)) { + if (unlikely(OPTION_COMP & *options)) { + *ptr++ = htonl((TCPOPT_EXP << 24) | + (TCPOLEN_EXP_COMP_BASE << 16) | + (TCPOPT_COMP_MAGIC)); + } + } +#endif +} + struct tcp_out_options { u16 options; /* bit field of OPTION_* */ u16 mss; /* 0 to disable */ @@ -711,6 +725,8 @@ static void tcp_options_write(struct tcphdr *th, struct tcp_sock *tp, smc_options_write(ptr, &options); mptcp_options_write(th, ptr, tp, opts); + + comp_options_write(ptr, &options); } static void smc_set_option(const struct tcp_sock *tp, @@ -729,6 +745,39 @@ static void smc_set_option(const struct tcp_sock *tp, #endif } +static void comp_set_option(struct sock *sk, + struct tcp_out_options *opts, + unsigned int *remaining) +{ +#if IS_ENABLED(CONFIG_TCP_COMP) + if (static_branch_unlikely(&tcp_have_comp)) { + if (tcp_syn_comp_enabled(sk)) { + if (*remaining >= TCPOLEN_EXP_COMP_BASE) { + opts->options |= OPTION_COMP; + *remaining -= TCPOLEN_EXP_COMP_BASE; + } + } + } +#endif +} + +static void comp_set_option_cond(struct sock *sk, + const struct inet_request_sock *ireq, + struct tcp_out_options *opts, + unsigned int *remaining) +{ +#if IS_ENABLED(CONFIG_TCP_COMP) + if (static_branch_unlikely(&tcp_have_comp)) { + if (tcp_synack_comp_enabled(sk, ireq)) { + if (*remaining >= TCPOLEN_EXP_COMP_BASE) { + opts->options |= OPTION_COMP; + *remaining -= TCPOLEN_EXP_COMP_BASE; + } + } + } +#endif +} + static void smc_set_option_cond(const struct tcp_sock *tp, const struct inet_request_sock *ireq, struct tcp_out_options *opts, @@ -830,6 +879,7 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb, } smc_set_option(tp, opts, &remaining); + comp_set_option(sk, opts, &remaining); if (sk_is_mptcp(sk)) { unsigned int size; @@ -910,6 +960,8 @@ static unsigned int tcp_synack_options(const struct sock *sk, smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining); + comp_set_option_cond((struct sock *)sk, ireq, opts, &remaining); + bpf_skops_hdr_opt_len((struct sock *)sk, skb, req, syn_skb, synack_type, opts, &remaining); diff --git a/net/ipv6/syncookies.c b/net/ipv6/syncookies.c index 8698b49dfc8d..75652826abbd 100644 --- a/net/ipv6/syncookies.c +++ b/net/ipv6/syncookies.c @@ -218,6 +218,10 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb) if (IS_ENABLED(CONFIG_SMC)) ireq->smc_ok = 0; +#if IS_ENABLED(CONFIG_TCP_COMP) + ireq->comp_ok = 0; +#endif + /* * We need to lookup the dst_entry to get the correct window size. * This is taken from tcp_v6_syn_recv_sock. Somebody please enlighten -- 2.34.1

2 1

[PATCH OLK-6.6 0/1] arm64: Add non nmi ipi backtrace support
by Liao Chen 03 Jan '24

03 Jan '24

From: linruizhe <linruizhe(a)huawei.com> Dependencies: CONFIG_DETECT_HUNG_TASK=y CONFIG_WATCHDOG=y Commandline Params: gic-version=3 irqchip.gicv3_pseudo_nmi=1 kgdboc=ttyAMA0,115200 Chen Jiahao (1): arm64: Fix the ipi backtrace warning when softlockup arch/arm64/kernel/ipi_nmi.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) Li Zhengyu (1): arm64: Add non nmi ipi backtrace support arch/arm64/kernel/ipi_nmi.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) -- 2.25.1

2 2

[PATCH OLK-6.6] ARM: spectre-v2: turn off the mitigation via boot cmdline param
by Yi Yang 03 Jan '24

03 Jan '24

From: "GONG, Ruiqi" <gongruiqi1(a)huawei.com> hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I8SE4Y ------------------------------------------------- We enable spectre mitigation by default for ARM32, which may cause performance regression. To offer an option to turn off this feature, implement a cmdline parameter 'nospectre_v2' compatible with mainline, which sets up a switch to skip invalidating BTB/icache for A9/A15 in context switching and user abort. Signed-off-by: GONG, Ruiqi <gongruiqi1(a)huawei.com> Signed-off-by: Yi Yang <yiyang13(a)huawei.com> --- arch/arm/include/asm/system_misc.h | 1 + arch/arm/mm/proc-v7-bugs.c | 18 ++++++++++++++++++ arch/arm/mm/proc-v7.S | 9 +++++++++ 3 files changed, 28 insertions(+) diff --git a/arch/arm/include/asm/system_misc.h b/arch/arm/include/asm/system_misc.h index 98b37340376b..4ca02f1927f7 100644 --- a/arch/arm/include/asm/system_misc.h +++ b/arch/arm/include/asm/system_misc.h @@ -36,6 +36,7 @@ static inline void harden_branch_predictor(void) #define UDBG_BUS (1 << 4) extern unsigned int user_debug; +extern int nospectre_v2; #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm/mm/proc-v7-bugs.c b/arch/arm/mm/proc-v7-bugs.c index 8bc7a2d6d6c7..b59c4d8ea762 100644 --- a/arch/arm/mm/proc-v7-bugs.c +++ b/arch/arm/mm/proc-v7-bugs.c @@ -9,6 +9,19 @@ #include <asm/spectre.h> #include <asm/system_misc.h> +/* + * 32-bit ARM spectre hardening, enabled by default, can be disabled via boot + * cmdline param 'nospectre_v2' to avoid performance regression. + */ +int nospectre_v2 __read_mostly; + +static int __init nospectre_v2_setup(char *str) +{ + nospectre_v2 = 1; + return 0; +} +early_param("nospectre_v2", nospectre_v2_setup); + #ifdef CONFIG_ARM_PSCI static int __maybe_unused spectre_v2_get_cpu_fw_mitigation_state(void) { @@ -118,6 +131,11 @@ static void cpu_v7_spectre_v2_init(void) { unsigned int state, method = 0; + if (nospectre_v2) { + pr_info_once("Spectre v2: hardening is disabled\n"); + return; + } + switch (read_cpuid_part()) { case ARM_CPU_PART_CORTEX_A8: case ARM_CPU_PART_CORTEX_A9: diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S index 193c7aeb6703..d1bbbb5dae7d 100644 --- a/arch/arm/mm/proc-v7.S +++ b/arch/arm/mm/proc-v7.S @@ -113,14 +113,23 @@ ENTRY(cpu_v7_hvc_switch_mm) b cpu_v7_switch_mm ENDPROC(cpu_v7_hvc_switch_mm) #endif + ENTRY(cpu_v7_iciallu_switch_mm) + ldr_l r3, nospectre_v2 + cmp r3, #1 + beq 1f mov r3, #0 mcr p15, 0, r3, c7, c5, 0 @ ICIALLU +1: b cpu_v7_switch_mm ENDPROC(cpu_v7_iciallu_switch_mm) ENTRY(cpu_v7_bpiall_switch_mm) + ldr_l r3, nospectre_v2 + cmp r3, #1 + beq 1f mov r3, #0 mcr p15, 0, r3, c7, c5, 6 @ flush BTAC/BTB +1: b cpu_v7_switch_mm ENDPROC(cpu_v7_bpiall_switch_mm) -- 2.25.1

2 1

[PATCH OpenEuler-1.0-LTS 0/3] Fix bugs from LTS patches
by Zhang Zekun 03 Jan '24

03 Jan '24

Bugfix from linux stable patches Mukesh Ojha (2): devcoredump : Serialize devcd_del work devcoredump: Send uevent once devcd is ready Saravana Kannan (1): driver core: Release all resources during unbind before updating device links drivers/base/dd.c | 4 +- drivers/base/devcoredump.c | 86 +++++++++++++++++++++++++++++++++++++- 2 files changed, 86 insertions(+), 4 deletions(-) -- 2.17.1

2 4

[PATCH OLK-6.6 000/123] arm_mpam: support mpam in olk_6.6
by Zeng Heng 02 Jan '24

02 Jan '24

Amit Singh Tomar (1): fs/resctrl: Remove the limit on the number of CLOSID Babu Moger (9): x86/resctrl: Add multiple tasks to the resctrl group at once x86/resctrl: Simplify rftype flag definitions x86/resctrl: Rename rftype flags for consistency x86/resctrl: Unwind properly from rdt_enable_ctx() x86/resctrl: Move default group file creation to mount x86/resctrl: Introduce "-o debug" mount option x86/resctrl: Display CLOSID for resource group x86/resctrl: Add support for the files of MON groups only x86/resctrl: Display RMID of resource group Fenghua Yu (1): x86/resctrl: Add sparse_masks file in info James Morse (104): tick/nohz: Move tick_nohz_full_mask declaration outside the #ifdef x86/resctrl: kfree() rmid_ptrs from resctrl_exit() x86/resctrl: Create helper for RMID allocation and mondata dir creation x86/resctrl: Move rmid allocation out of mkdir_rdt_prepare() x86/resctrl: Track the closid with the rmid x86/resctrl: Access per-rmid structures by index x86/resctrl: Allow RMID allocation to be scoped by CLOSID x86/resctrl: Track the number of dirty RMID a CLOSID has x86/resctrl: Use __set_bit()/__clear_bit() instead of open coding x86/resctrl: Allocate the cleanest CLOSID by searching closid_num_dirty_rmid x86/resctrl: Move CLOSID/RMID matching and setting to use helpers x86/resctrl: Add cpumask_any_housekeeping() for limbo/overflow x86/resctrl: Queue mon_event_read() instead of sending an IPI x86/resctrl: Allow resctrl_arch_rmid_read() to sleep x86/resctrl: Allow arch to allocate memory needed in resctrl_arch_rmid_read() x86/resctrl: Make resctrl_mounted checks explicit x86/resctrl: Move alloc/mon static keys into helpers x86/resctrl: Make rdt_enable_key the arch's decision to switch x86/resctrl: Add helpers for system wide mon/alloc capable x86/resctrl: Add CPU online callback for resctrl work x86/resctrl: Allow overflow/limbo handlers to be scheduled on any-but cpu x86/resctrl: Add CPU offline callback for resctrl work x86/resctrl: Move domain helper migration into resctrl_offline_cpu() x86/resctrl: Separate arch and fs resctrl locks x86/resctrl: Add a helper to avoid reaching into the arch code resource list x86/resctrl: Move ctrlval string parsing links away from the arch code x86/resctrl: Add helper for setting CPU default properties x86/resctrl: Remove rdtgroup from update_cpu_closid_rmid() x86/resctrl: Export resctrl fs's init function x86/resctrl: Wrap resctrl_arch_find_domain() around rdt_find_domain() x86/resctrl: Move resctrl types to a separate header x86/resctrl: Add a resctrl helper to reset all the resources x86/resctrl: Move monitor init work to a resctrl init call x86/resctrl: Move monitor exit work to a restrl exit call x86/resctrl: Move max_{name,data}_width into resctrl code x86/resctrl: Stop using the for_each_*_rdt_resource() walkers x86/resctrl: Export the is_mbm_*_enabled() helpers to asm/resctrl.h x86/resctrl: Add resctrl_arch_is_evt_configurable() to abstract BMEC x86/resctrl: Change mon_event_config_{read,write}() to be arch helpers x86/resctrl: Allow resctrl_arch_mon_event_config_write() to return an error x86/resctrl: Allow an architecture to disable pseudo lock x86/resctrl: Make prefetch_disable_bits belong to the arch code x86/resctrl: Make resctrl_arch_pseudo_lock_fn() take a plr x86/resctrl: Move thread_throttle_mode_init() to be managed by resctrl x86/resctrl: Move get_config_index() to a header x86/resctrl: Claim get_domain_from_cpu() for resctrl x86/resctrl: Describe resctrl's bitmap size assumptions x86/resctrl: Drop __init/__exit on assorted symbols fs/resctrl: Add boiler plate for external resctrl code x86/resctrl: Move the filesystem bits to headers visible to fs/resctrl x86/resctrl: Move the filesystem portions of resctrl to live in '/fs/' arm64: head.S: Initialise MPAM EL2 registers and disable traps arm64: cpufeature: discover CPU support for MPAM KVM: arm64: Fix missing traps of guest accesses to the MPAM registers KVM: arm64: Disable MPAM visibility by default, and handle traps arm64: mpam: Context switch the MPAM registers untested: KVM: arm64: Force guest EL1 to use user-space's partid configuration ACPI / PPTT: Provide a helper to walk processor containers ACPI / PPTT: Find PPTT cache level by ID ACPI / PPTT: Add a helper to fill a cpumask from a processor container ACPI / PPTT: Add a helper to fill a cpumask from a cache_id cacheinfo: Expose the code to generate a cache-id from a device_node drivers: base: cacheinfo: Add helper to find the cache size from cpu+level ACPI / MPAM: Parse the MPAM table arm_mpam: Add probe/remove for mpam msc driver and kbuild boiler plate arm_mpam: Add the class and component structures for ris firmware described arm_mpam: Add MPAM MSC register layout definitions arm_mpam: Add cpuhp callbacks to probe MSC hardware arm_mpam: Probe MSCs to find the supported partid/pmg values arm_mpam: Probe the hardware features resctrl supports arm_mpam: Merge supported features during mpam_enable() into mpam_class arm_mpam: Reset MSC controls from cpu hp callbacks arm_mpam: Add a helper to touch an MSC from any CPU arm_mpam: Extend reset logic to allow devices to be reset any time arm_mpam: Register and enable IRQs arm_mpam: Use the arch static key to indicate when mpam is enabled arm_mpam: Allow configuration to be applied and restored during cpu online arm_mpam: Probe and reset the rest of the features arm_mpam: Add helpers to allocate monitors arm_mpam: Add mpam_msmon_read() to read monitor value arm_mpam: Track bandwidth counter state for overflow and power management arm_mpam: Add helper to reset saved mbwu state arm_mpam: resctrl: Add boilerplate cpuhp and domain allocation arm_mpam: resctrl: Pick the caches we will use as resctrl resources arm_mpam: resctrl: Pick a value for num_rmid arm_mpam: resctrl: Implement resctrl_arch_reset_resources() arm_mpam: resctrl: Add resctrl_arch_get_config() arm_mpam: resctrl: Implement helpers to update configuration arm_mpam: resctrl: Add CDP emulation arm64: mpam: Add helpers to change a tasks and cpu mpam partid/pmg values arm_mpam: resctrl: Add rmid index helpers untested: arm_mpam: resctrl: Add support for MB resource untested: arm_mpam: resctrl: Add support for mbm counters arm_mpam: resctrl: Allow resctrl to allocate monitors arm_mpam: resctrl: Add resctrl_arch_rmid_read() and resctrl_arch_reset_rmid() untested: arm_mpam: resctrl: Allow monitors to be configured arm_mpam: resctrl: Add empty definitions for pseudo lock arm_mpam: resctrl: Add empty definitions for fine-grained enables arm_mpam: resctrl: Add dummy definition for free running counters arm64: mpam: Select ARCH_HAS_CPU_RESCTRL perf/arm-cmn: Stop claiming all the resources arm_mpam: resctrl: Tell resctrl about cpu/domain online/offline arm_mpam: resctrl: Call resctrl_exit() in the event of errors arm_mpam: resctrl: Update the rmid reallocation limit Maciej Wieczor-Retman (3): x86/resctrl: Fix remaining kernel-doc warnings x86/resctrl: Rename arch_has_sparse_bitmaps x86/resctrl: Enable non-contiguous CBMs in Intel CAT Rob Herring (3): cacheinfo: Allow for >32-bit cache 'id' cacheinfo: Set cache 'id' based on DT data dt-bindings: arm: Add MPAM MSC binding Rohit Mathew (2): arm_mpam: Probe for long/lwd mbwu counters arm_mpam: Use long MBWU counters if supported .../arch/arm64/cpu-feature-registers.rst | 2 + Documentation/arch/x86/resctrl.rst | 22 +- .../devicetree/bindings/arm/arm,mpam-msc.yaml | 227 + MAINTAINERS | 2 + arch/Kconfig | 8 + arch/arm64/Kconfig | 22 +- arch/arm64/include/asm/cpu.h | 1 + arch/arm64/include/asm/cpufeature.h | 13 + arch/arm64/include/asm/el2_setup.h | 16 + arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/mpam.h | 167 + arch/arm64/include/asm/resctrl.h | 1 + arch/arm64/include/asm/sysreg.h | 8 + arch/arm64/include/asm/thread_info.h | 3 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/cpufeature.c | 68 + arch/arm64/kernel/cpuinfo.c | 4 + arch/arm64/kernel/image-vars.h | 5 + arch/arm64/kernel/mpam.c | 23 + arch/arm64/kernel/process.c | 7 + arch/arm64/kvm/hyp/include/hyp/switch.h | 32 + arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 27 + arch/arm64/kvm/hyp/nvhe/switch.c | 11 + arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 1 + arch/arm64/kvm/sys_regs.c | 70 +- arch/arm64/tools/cpucaps | 1 + arch/arm64/tools/sysreg | 33 + arch/x86/Kconfig | 5 +- arch/x86/include/asm/resctrl.h | 131 +- arch/x86/kernel/cpu/resctrl/core.c | 212 +- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 476 +- arch/x86/kernel/cpu/resctrl/internal.h | 394 +- arch/x86/kernel/cpu/resctrl/monitor.c | 594 +-- arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 1123 +---- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 3916 +--------------- drivers/acpi/arm64/Kconfig | 3 + drivers/acpi/arm64/Makefile | 2 + drivers/acpi/arm64/mpam.c | 369 ++ drivers/acpi/pptt.c | 278 +- drivers/acpi/tables.c | 2 +- drivers/base/cacheinfo.c | 41 +- drivers/perf/arm-cmn.c | 12 +- drivers/platform/Kconfig | 2 + drivers/platform/Makefile | 1 + drivers/platform/mpam/Kconfig | 8 + drivers/platform/mpam/Makefile | 1 + drivers/platform/mpam/mpam_devices.c | 2422 ++++++++++ drivers/platform/mpam/mpam_internal.h | 576 +++ drivers/platform/mpam/mpam_resctrl.c | 1209 +++++ fs/Kconfig | 1 + fs/Makefile | 1 + fs/resctrl/Kconfig | 23 + fs/resctrl/Makefile | 1 + fs/resctrl/ctrlmondata.c | 528 +++ fs/resctrl/internal.h | 313 ++ fs/resctrl/monitor.c | 859 ++++ fs/resctrl/psuedo_lock.c | 1134 +++++ fs/resctrl/rdtgroup.c | 4015 +++++++++++++++++ include/linux/acpi.h | 17 + include/linux/arm_mpam.h | 105 + include/linux/cacheinfo.h | 30 +- include/linux/resctrl.h | 203 +- include/linux/resctrl_types.h | 98 + include/linux/tick.h | 9 +- 64 files changed, 13418 insertions(+), 6472 deletions(-) create mode 100644 Documentation/devicetree/bindings/arm/arm,mpam-msc.yaml create mode 100644 arch/arm64/include/asm/mpam.h create mode 100644 arch/arm64/include/asm/resctrl.h create mode 100644 arch/arm64/kernel/mpam.c create mode 100644 drivers/acpi/arm64/mpam.c create mode 100644 drivers/platform/mpam/Kconfig create mode 100644 drivers/platform/mpam/Makefile create mode 100644 drivers/platform/mpam/mpam_devices.c create mode 100644 drivers/platform/mpam/mpam_internal.h create mode 100644 drivers/platform/mpam/mpam_resctrl.c create mode 100644 fs/resctrl/Kconfig create mode 100644 fs/resctrl/Makefile create mode 100644 fs/resctrl/ctrlmondata.c create mode 100644 fs/resctrl/internal.h create mode 100644 fs/resctrl/monitor.c create mode 100644 fs/resctrl/psuedo_lock.c create mode 100644 fs/resctrl/rdtgroup.c create mode 100644 include/linux/arm_mpam.h create mode 100644 include/linux/resctrl_types.h -- 2.25.1

2 124

[PATCH OLK-6.6 0/8] backport etmem feature from OLK-5.10
by Yuchen Tang 02 Jan '24

02 Jan '24

Kemeng Shi (2): x86: hugepage: use nt copy hugepage to AEP in x86 ugepage: add sysctl for hugepage alloc and mig Yuchen Tang (6): etmem: add etmem scan feature to openEuler etmem: add etmem swap feature to openEuler mm: introduce and export pgdat peer_node mm: alloc pages from pmem's peer node etmem: add original kernel swap enabled options etmem: add swapcache reclaim to etmem arch/arm64/configs/openeuler_defconfig | 3 + arch/x86/configs/openeuler_defconfig | 3 + arch/x86/include/asm/page_64.h | 9 + arch/x86/lib/Makefile | 1 + arch/x86/lib/copy_highpages.c | 107 ++ arch/x86/lib/copy_page_64.S | 73 ++ drivers/acpi/numa/srat.c | 7 + drivers/base/node.c | 15 + fs/proc/Makefile | 3 + fs/proc/base.c | 8 + fs/proc/etmem_proc.c | 216 ++++ fs/proc/etmem_scan.c | 1440 ++++++++++++++++++++++++ fs/proc/etmem_scan.h | 149 +++ fs/proc/etmem_swap.c | 283 +++++ fs/proc/internal.h | 4 + include/linux/etmem.h | 105 ++ include/linux/highmem.h | 17 + include/linux/mm.h | 4 + include/linux/mm_types.h | 6 + include/linux/mmzone.h | 14 + include/linux/swap.h | 2 +- include/uapi/asm-generic/mman-common.h | 4 + mm/Kconfig | 33 + mm/Makefile | 1 + mm/etmem.c | 475 ++++++++ mm/hugetlb.c | 9 + mm/madvise.c | 16 + mm/mm_init.c | 4 + mm/pagewalk.c | 3 + mm/sparse-vmemmap.c | 10 + mm/swap_state.c | 4 + mm/util.c | 5 + mm/vmscan.c | 21 + virt/kvm/kvm_main.c | 6 + 34 files changed, 3059 insertions(+), 1 deletion(-) create mode 100644 arch/x86/lib/copy_highpages.c create mode 100644 fs/proc/etmem_proc.c create mode 100644 fs/proc/etmem_scan.c create mode 100644 fs/proc/etmem_scan.h create mode 100644 fs/proc/etmem_swap.c create mode 100644 include/linux/etmem.h create mode 100644 mm/etmem.c -- 2.33.0

3 10

[PATCH OLK-6.6 0/7] introduce smart_grid zone
by Yipeng Zou 02 Jan '24

02 Jan '24

The patch sets include two parts: 1. patch 1~3: Rebase smart_grid from OLK-5.10 to OLK-6.6 2. patch 4~7: introduce smart_grid zone qos and cpufreq Hui Tang (1): sched: Introduce smart grid scheduling strategy for cfs Wang ShaoBo (2): sched: smart grid: init sched_grid_qos structure on QOS purpose config: enable CONFIG_QOS_SCHED_SMART_GRID by default Yipeng Zou (4): sched: introduce smart grid qos zone smart_grid: introduce /proc/pid/smart_grid_level smart_grid: introduce smart_grid_strategy_ctrl sysctl smart_grid: cpufreq: introduce smart_grid cpufreq control arch/arm64/configs/openeuler_defconfig | 1 + arch/x86/configs/openeuler_defconfig | 1 + drivers/cpufreq/cpufreq.c | 232 ++++++++++++ fs/proc/array.c | 16 + fs/proc/base.c | 78 ++++ include/linux/cpufreq.h | 11 + include/linux/sched.h | 17 + include/linux/sched/grid_qos.h | 135 +++++++ include/linux/sched/sysctl.h | 5 + init/Kconfig | 13 + kernel/fork.c | 12 + kernel/sched/Makefile | 1 + kernel/sched/core.c | 181 +++++++++ kernel/sched/fair.c | 503 ++++++++++++++++++++++++- kernel/sched/grid/Makefile | 2 + kernel/sched/grid/internal.h | 6 + kernel/sched/grid/power.c | 27 ++ kernel/sched/grid/qos.c | 279 ++++++++++++++ kernel/sched/grid/stat.c | 47 +++ kernel/sched/sched.h | 48 ++- kernel/sysctl.c | 23 ++ mm/mempolicy.c | 11 +- 22 files changed, 1638 insertions(+), 11 deletions(-) create mode 100644 include/linux/sched/grid_qos.h create mode 100644 kernel/sched/grid/Makefile create mode 100644 kernel/sched/grid/internal.h create mode 100644 kernel/sched/grid/power.c create mode 100644 kernel/sched/grid/qos.c create mode 100644 kernel/sched/grid/stat.c -- 2.34.1

2 8

[PATCH OLK-6.6 0/2] lib/clear_user: ensure loop in __arch_clear_user cache-aligned v2
by Yang Yingliang 02 Jan '24

02 Jan '24

We must ensure that the following four instructions are cache-aligned. Otherwise, it will cause problems with the performance of libMicro pread. 1: # uao_user_alternative 9f, str, sttr, xzr, x0, 8 str xzr, [x0], #8 nop subs x1, x1, #8 b.pl 1b with this patch: prc thr usecs/call samples errors cnt/samp size pread_z100 1 1 5.88400 807 0 1 102400 The result of pread can range from 5 to 9 depending on the alignment performance of this function. Cheng Jian (1): lib/clear_user: ensure loop in __arch_clear_user cache-aligned v2 Yang Yingliang (1): config: enable CONFIG_CLEAR_USER_WORKAROUND by default arch/arm64/Kconfig | 7 +++++++ arch/arm64/configs/openeuler_defconfig | 1 + arch/arm64/lib/clear_user.S | 3 +++ 3 files changed, 11 insertions(+) -- 2.25.1

2 3