[PATCH OLK-6.6 0/3] fuse: support sysfs api for flush and
Support sysfs api for flush and resend pending requests, note that OLK-6.6 already supports request resend through fuse notify. Long Li (1): fuse: fix issues in fuse_resend_pqueue() implementation Ma Jie Yue (1): fuse: Introduce sysfs API for flushing pending requests Peng Tao (1): fuse: Introduce sysfs API for resend pending requests fs/fuse/control.c | 40 ++++++++++++++++++++++++++++++++++++++++ fs/fuse/dev.c | 39 ++++++++++++++++++++++++++++++++++++++- fs/fuse/fuse_i.h | 8 +++++++- 3 files changed, 85 insertions(+), 2 deletions(-) -- 2.39.2
From: Ma Jie Yue <majieyue@linux.alibaba.com> anolis inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/ICFOA8 CVE: NA -------------------------------- Commit a07f55b5f686677333b96a8c188d6c6c6cd6e006 OpenAnolis In certain scenarios, to enhance availability, a new FUSE userspace daemon is launched to continue providing services after a crash of the previous daemon. This is achieved by performing an fd takeover from a dedicated watchdog daemon. However, if some inflight requests are lost during the crash, the application becomes stuck as it never receives a reply. To address this issue, this commit introduces a sysfs API that allows for flushing these pending requests after a daemon crash, prior to initiating the recovery procedure. Instead of remaining stuck, the flush operation returns an error to the application, causing the inflight requests to fail quickly. While returning an error may not be suitable for all scenarios, we have also submitted a separate patch to enable the ability to resend the inflight requests. Signed-off-by: Ma Jie Yue <majieyue@linux.alibaba.com> Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com> Reviewed-by: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com> Acked-by: Caspar Zhang <caspar@linux.alibaba.com> Signed-off-by: Hao Xu <haoxu@linux.alibaba.com> Signed-off-by: Zhao Chen <winters.zc@antgroup.com> Signed-off-by: Long Li <leo.lilong@huawei.com> --- fs/fuse/control.c | 20 ++++++++++++++++++++ fs/fuse/dev.c | 37 +++++++++++++++++++++++++++++++++++++ fs/fuse/fuse_i.h | 5 ++++- 3 files changed, 61 insertions(+), 1 deletion(-) diff --git a/fs/fuse/control.c b/fs/fuse/control.c index 1bf928e277fe..c0baf79977eb 100644 --- a/fs/fuse/control.c +++ b/fs/fuse/control.c @@ -44,6 +44,18 @@ static ssize_t fuse_conn_abort_write(struct file *file, const char __user *buf, return count; } +static ssize_t fuse_conn_flush_write(struct file *file, const char __user *buf, + size_t count, loff_t *ppos) +{ + struct fuse_conn *fc = fuse_ctl_file_conn_get(file); + + if (fc) { + fuse_flush_pq(fc); + fuse_conn_put(fc); + } + return count; +} + static ssize_t fuse_conn_waiting_read(struct file *file, char __user *buf, size_t len, loff_t *ppos) { @@ -186,6 +198,12 @@ static const struct file_operations fuse_ctl_abort_ops = { .llseek = no_llseek, }; +static const struct file_operations fuse_ctl_flush_ops = { + .open = nonseekable_open, + .write = fuse_conn_flush_write, + .llseek = no_llseek, +}; + static const struct file_operations fuse_ctl_waiting_ops = { .open = nonseekable_open, .read = fuse_conn_waiting_read, @@ -270,6 +288,8 @@ int fuse_ctl_add_conn(struct fuse_conn *fc) NULL, &fuse_ctl_waiting_ops) || !fuse_ctl_add_dentry(parent, fc, "abort", S_IFREG | 0200, 1, NULL, &fuse_ctl_abort_ops) || + !fuse_ctl_add_dentry(parent, fc, "flush", S_IFREG | 0200, 1, + NULL, &fuse_ctl_flush_ops) || !fuse_ctl_add_dentry(parent, fc, "max_background", S_IFREG | 0600, 1, NULL, &fuse_conn_max_background_ops) || !fuse_ctl_add_dentry(parent, fc, "congestion_threshold", diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 9d6158d2da46..938c1e9f257b 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -2361,6 +2361,43 @@ int fuse_dev_release(struct inode *inode, struct file *file) } EXPORT_SYMBOL_GPL(fuse_dev_release); +/* + * Flush all pending processing requests. + * + * The failover procedure reuses the fuse_conn after a userspace crash and + * recovery. However, the requests in the processing queue will never receive a + * reply, causing the application to become stuck indefinitely. + * + * To resolve this issue, we need to flush these requests using the sysfs API. + * We only flush the requests in the processing queue, as these requests have + * already been sent to userspace. First, we dequeue the request from the + * processing queue, and then we call request_end to finalize it. + */ +void fuse_flush_pq(struct fuse_conn *fc) +{ + struct fuse_dev *fud; + LIST_HEAD(to_end); + unsigned int i; + + spin_lock(&fc->lock); + if (!fc->connected) { + spin_unlock(&fc->lock); + return; + } + list_for_each_entry(fud, &fc->devices, entry) { + struct fuse_pqueue *fpq = &fud->pq; + + spin_lock(&fpq->lock); + WARN_ON(!list_empty(&fpq->io)); + for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) + list_splice_init(&fpq->processing[i], &to_end); + spin_unlock(&fpq->lock); + } + spin_unlock(&fc->lock); + + end_requests(&to_end); +} + static int fuse_dev_fasync(int fd, struct file *file, int on) { struct fuse_dev *fud = fuse_get_dev(file); diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 665e89d8ea5b..346034ef02bd 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -46,7 +46,7 @@ #define FUSE_NAME_MAX 1024 /** Number of dentries for each connection in the control filesystem */ -#define FUSE_CTL_NUM_DENTRIES 5 +#define FUSE_CTL_NUM_DENTRIES 6 /** Maximum of max_pages received in init_out */ extern unsigned int fuse_max_pages_limit; @@ -1200,6 +1200,9 @@ void fuse_request_end(struct fuse_req *req); void fuse_abort_conn(struct fuse_conn *fc); void fuse_wait_aborted(struct fuse_conn *fc); +/* Flush all requests in processing queue */ +void fuse_flush_pq(struct fuse_conn *fc); + /** * Invalidate inode attributes */ -- 2.39.2
From: Peng Tao <tao.peng@linux.alibaba.com> anolis inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/ICFOA8 CVE: NA -------------------------------- commit 017eaddf79d79778d3ba4af5ed1076a1e9558151 OpenAnolis When a FUSE daemon panics and fails over, we want to reuse the existing FUSE connection and avoid affecting applications as little as possible. During FUSE daemon failover, the FUSE processing queue requests are waiting for replies from user space daemon that never come back and applications would stuck forever. Besides flushing the processing queue requests like being done in fuse_flush_pq(), we can also resend these requests to user space daemon so that they can be processed properly again. Such strategy can only be done for idempotent requests or if the user space daemon takes good care to record and avoid processing duplicated non-idempotent requests, otherwise there can be consistency issues. We trust users to know what they are doing by calling writing to this interface. Signed-off-by: Peng Tao <tao.peng@linux.alibaba.com> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com> Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com> Reviewed-by: Jeffle Xu <jefflexu@linux.alibaba.com> Acked-by: Gao Xiang <hsiangkao@linux.alibaba.com> Signed-off-by: Zhao Chen <winters.zc@antgroup.com> Signed-off-by: Long Li <leo.lilong@huawei.com> --- fs/fuse/control.c | 20 ++++++++++++++++ fs/fuse/dev.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++ fs/fuse/fuse_i.h | 5 +++- 3 files changed, 85 insertions(+), 1 deletion(-) diff --git a/fs/fuse/control.c b/fs/fuse/control.c index c0baf79977eb..8ae0f8625e59 100644 --- a/fs/fuse/control.c +++ b/fs/fuse/control.c @@ -56,6 +56,18 @@ static ssize_t fuse_conn_flush_write(struct file *file, const char __user *buf, return count; } +static ssize_t fuse_conn_resend_write(struct file *file, const char __user *buf, + size_t count, loff_t *ppos) +{ + struct fuse_conn *fc = fuse_ctl_file_conn_get(file); + + if (fc) { + fuse_resend_pqueue(fc); + fuse_conn_put(fc); + } + return count; +} + static ssize_t fuse_conn_waiting_read(struct file *file, char __user *buf, size_t len, loff_t *ppos) { @@ -204,6 +216,12 @@ static const struct file_operations fuse_ctl_flush_ops = { .llseek = no_llseek, }; +static const struct file_operations fuse_ctl_resend_ops = { + .open = nonseekable_open, + .write = fuse_conn_resend_write, + .llseek = no_llseek, +}; + static const struct file_operations fuse_ctl_waiting_ops = { .open = nonseekable_open, .read = fuse_conn_waiting_read, @@ -290,6 +308,8 @@ int fuse_ctl_add_conn(struct fuse_conn *fc) NULL, &fuse_ctl_abort_ops) || !fuse_ctl_add_dentry(parent, fc, "flush", S_IFREG | 0200, 1, NULL, &fuse_ctl_flush_ops) || + !fuse_ctl_add_dentry(parent, fc, "resend", S_IFREG | 0200, 1, + NULL, &fuse_ctl_resend_ops) || !fuse_ctl_add_dentry(parent, fc, "max_background", S_IFREG | 0600, 1, NULL, &fuse_conn_max_background_ops) || !fuse_ctl_add_dentry(parent, fc, "congestion_threshold", diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 938c1e9f257b..d38db9ed4f35 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -2398,6 +2398,67 @@ void fuse_flush_pq(struct fuse_conn *fc) end_requests(&to_end); } +/** + * Resend all processing queue requests. + * + * When a FUSE daemon panics and fails over, we want to reuse the existing FUSE + * connection and avoid affecting applications as little as possible. During + * FUSE daemon failover, the FUSE processing queue requests are waiting for + * replies from user space daemon that never come back and applications would + * stuck forever. + * + * Besides flushing the processing queue requests like being done in fuse_flush_pq(), + * we can also resend these requests to user space daemon so that they can be + * processed properly again. Such strategy can only be done for idempotent requests + * or if the user space daemon takes good care to record and avoid processing + * duplicated non-idempotent requests. + */ +void fuse_resend_pqueue(struct fuse_conn *fc) +{ + struct fuse_dev *fud; + struct fuse_req *req, *next; + struct fuse_iqueue *fiq = &fc->iq; + LIST_HEAD(to_queue); + unsigned int i; + + spin_lock(&fc->lock); + if (!fc->connected) { + spin_unlock(&fc->lock); + return; + } + + list_for_each_entry(fud, &fc->devices, entry) { + struct fuse_pqueue *fpq = &fud->pq; + + spin_lock(&fpq->lock); + /* + * Move requests for resend, including two parts: unlocked + * requests in io list, and all requests in pending queue. + */ + list_for_each_entry_safe(req, next, &fpq->io, list) { + spin_lock(&req->waitq.lock); + if (!test_bit(FR_LOCKED, &req->flags)) { + __fuse_get_request(req); + list_move(&req->list, &to_queue); + } + spin_unlock(&req->waitq.lock); + } + for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) + list_splice_tail_init(&fpq->processing[i], &to_queue); + spin_unlock(&fpq->lock); + } + spin_unlock(&fc->lock); + + list_for_each_entry_safe(req, next, &to_queue, list) { + __set_bit(FR_PENDING, &req->flags); + } + + spin_lock(&fiq->lock); + /* iq and pq requests are both oldest to newest */ + list_splice(&to_queue, &fiq->pending); + spin_unlock(&fiq->lock); +} + static int fuse_dev_fasync(int fd, struct file *file, int on) { struct fuse_dev *fud = fuse_get_dev(file); diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 346034ef02bd..a5f88c0bdeb9 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -46,7 +46,7 @@ #define FUSE_NAME_MAX 1024 /** Number of dentries for each connection in the control filesystem */ -#define FUSE_CTL_NUM_DENTRIES 6 +#define FUSE_CTL_NUM_DENTRIES 7 /** Maximum of max_pages received in init_out */ extern unsigned int fuse_max_pages_limit; @@ -1203,6 +1203,9 @@ void fuse_wait_aborted(struct fuse_conn *fc); /* Flush all requests in processing queue */ void fuse_flush_pq(struct fuse_conn *fc); +/* Resend all requests in processing queue so they can represent to userspace */ +void fuse_resend_pqueue(struct fuse_conn *fc); + /** * Invalidate inode attributes */ -- 2.39.2
hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/ICFOA8 CVE: NA -------------------------------- Fix several issues in fuse_resend_pqueue(): 1. Set FR_PENDING flag atomically when resending request 2. Clear FR_SENT flag before resending 3. Check for aborted connection before adding requests to pending list 4. Wake up fiq->waitq after adding request back to pending queue Since kernel has merged 97f30876c943 ("fuse: Introduce a new notification type for resend pending requests") which already supports request resend functionality but not triggered via sysfs, simplify the code by replacing fuse_resend_pqueue() with fuse_resend(). This helps maintain code simplicity while ensuring correct request resend behavior. Signed-off-by: Long Li <leo.lilong@huawei.com> --- fs/fuse/control.c | 2 +- fs/fuse/dev.c | 63 +---------------------------------------------- fs/fuse/fuse_i.h | 2 +- 3 files changed, 3 insertions(+), 64 deletions(-) diff --git a/fs/fuse/control.c b/fs/fuse/control.c index 8ae0f8625e59..9c7ce06276b3 100644 --- a/fs/fuse/control.c +++ b/fs/fuse/control.c @@ -62,7 +62,7 @@ static ssize_t fuse_conn_resend_write(struct file *file, const char __user *buf, struct fuse_conn *fc = fuse_ctl_file_conn_get(file); if (fc) { - fuse_resend_pqueue(fc); + fuse_resend(fc); fuse_conn_put(fc); } return count; diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index d38db9ed4f35..98500f0f02ea 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -1860,7 +1860,7 @@ static int fuse_notify_retrieve(struct fuse_conn *fc, unsigned int size, * if the FUSE daemon takes careful measures to avoid processing duplicated * non-idempotent requests. */ -static void fuse_resend(struct fuse_conn *fc) +void fuse_resend(struct fuse_conn *fc) { struct fuse_dev *fud; struct fuse_req *req, *next; @@ -2398,67 +2398,6 @@ void fuse_flush_pq(struct fuse_conn *fc) end_requests(&to_end); } -/** - * Resend all processing queue requests. - * - * When a FUSE daemon panics and fails over, we want to reuse the existing FUSE - * connection and avoid affecting applications as little as possible. During - * FUSE daemon failover, the FUSE processing queue requests are waiting for - * replies from user space daemon that never come back and applications would - * stuck forever. - * - * Besides flushing the processing queue requests like being done in fuse_flush_pq(), - * we can also resend these requests to user space daemon so that they can be - * processed properly again. Such strategy can only be done for idempotent requests - * or if the user space daemon takes good care to record and avoid processing - * duplicated non-idempotent requests. - */ -void fuse_resend_pqueue(struct fuse_conn *fc) -{ - struct fuse_dev *fud; - struct fuse_req *req, *next; - struct fuse_iqueue *fiq = &fc->iq; - LIST_HEAD(to_queue); - unsigned int i; - - spin_lock(&fc->lock); - if (!fc->connected) { - spin_unlock(&fc->lock); - return; - } - - list_for_each_entry(fud, &fc->devices, entry) { - struct fuse_pqueue *fpq = &fud->pq; - - spin_lock(&fpq->lock); - /* - * Move requests for resend, including two parts: unlocked - * requests in io list, and all requests in pending queue. - */ - list_for_each_entry_safe(req, next, &fpq->io, list) { - spin_lock(&req->waitq.lock); - if (!test_bit(FR_LOCKED, &req->flags)) { - __fuse_get_request(req); - list_move(&req->list, &to_queue); - } - spin_unlock(&req->waitq.lock); - } - for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) - list_splice_tail_init(&fpq->processing[i], &to_queue); - spin_unlock(&fpq->lock); - } - spin_unlock(&fc->lock); - - list_for_each_entry_safe(req, next, &to_queue, list) { - __set_bit(FR_PENDING, &req->flags); - } - - spin_lock(&fiq->lock); - /* iq and pq requests are both oldest to newest */ - list_splice(&to_queue, &fiq->pending); - spin_unlock(&fiq->lock); -} - static int fuse_dev_fasync(int fd, struct file *file, int on) { struct fuse_dev *fud = fuse_get_dev(file); diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index a5f88c0bdeb9..ac101018bdba 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -1204,7 +1204,7 @@ void fuse_wait_aborted(struct fuse_conn *fc); void fuse_flush_pq(struct fuse_conn *fc); /* Resend all requests in processing queue so they can represent to userspace */ -void fuse_resend_pqueue(struct fuse_conn *fc); +void fuse_resend(struct fuse_conn *fc); /** * Invalidate inode attributes -- 2.39.2
反馈： 您发送到kernel@openeuler.org的补丁/补丁集，已成功转换为PR！ PR链接地址： https://gitee.com/openeuler/kernel/pulls/18237 邮件列表地址：https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/HED... FeedBack: The patch(es) which you have sent to kernel@openeuler.org mailing list has been converted to a pull request successfully! Pull request link: https://gitee.com/openeuler/kernel/pulls/18237 Mailing list address: https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/HED...
participants (2)
-
Long Li -
patchwork bot