Coredump appears when events are executed in a certain order.
1. The sdc and sdd are added as paths of d1. The sdc is added
firstly and d1->wait_for_udev be set to 1. Then sdd is added but
it just adds to pathvec because d1->wait_for_udev is 1.
2. The uev_remove_path(sdc) are executed. Because d1 has one
path, d1 will be flushed.
3. The uev_add_path(sdc) are executed. Now sdc and sdd will
both add to d1 in adopt_paths because sdd->wwid is d1. And
d1->wait_for_udev be set to 1.
4. …
[View More]The uev_remove_path(sdd) are executed. Because d1->wait_for_udev
is 1, domap doesn't executed and the major:minor of sdd(8:64)
still exist in kernel d1. The sdd in pathvec is deleted.
5. The uev_add_path(sdd) are executed. Now the sdd(8:64) is
a path of d2. And new sdd is added to pathvec.
6. The update_multipath_strings(d1) are executed. The d1 find
sdd in pathvec by dev_t(8:64)and sdd will be add to d1. Now
the pointer of sdd is store in paths of d1 and d2.
7. When NIC is down and a new path add to d1, sdd will be freed
in verify_paths (because sdd is actually a path of d2).
8. When d2->hwe is sdd->hwe and setup_map(d2) is called. Core
causes.
The path shouldn't be add to the mpp whose wwid is different
with its. Here we add wwid check in disassemble_map. This bug
has been solved in upstream in update_pathvec_from_dm. It is
a great change, so we fix it with huawei patch. This patch could
be discarded when multipath-tools update.
Signed-off-by:lixiaokeng<lixiaokeng(a)huawei.com>
---
libmultipath/dmparser.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/libmultipath/dmparser.c b/libmultipath/dmparser.c
index 0f370e9..99b57db 100644
--- a/libmultipath/dmparser.c
+++ b/libmultipath/dmparser.c
@@ -342,6 +342,16 @@ int disassemble_map(vector pathvec, char *params, struct multipath *mpp,
goto out1;
}
} else {
+ if (strlen(pp->wwid) && strlen(mpp->wwid) &&
+ strcmp(pp->wwid, mpp->wwid) != 0) {
+ condlog(0, "%s: path wwid is different with that of %s.\n", pp->dev_t, mpp->alias);
+ FREE(word);
+ for (k = 0; k < num_paths_args; k++) {
+ p += get_word(p, NULL);
+ }
+ continue;
+ }
+
if (!strlen(pp->wwid) &&
strlen(mpp->wwid))
strlcpy(pp->wwid, mpp->wwid,
--
[View Less]
When one iscsi device logs in and logs out with the "multipath -r"
executed at the same time, memory leak happens in multipathd
process.
The reason is following. When "multipath -r" is executed, the path
will be free in configure function. Before path_discovery executed,
iscsi device logs out. Then path_discovery will not find any path and
there is no path in the gvecs->pathvec. When map_discovery function
is executed, disassemble_map function will be called. Because
gvecs->pathvec->…
[View More]slot is empty and is_deamon is 1, a path will be
allocated and is not stored in gvecs->pathvec but store in
mpp->pg. But when the mpp is removed and freed by remove_map
function, the path will not be free and can't be find anymore.
The procedure details given as follows,
1."multipath -r" is executed
main
->child
->reconfigure
->configure
->path_discovery //after iscsi logout
->map_discovery
->update_multipath_table
->disassemble_map
->alloc_path
2.then "multipath -r" is executed again
main
->child
->reconfigure
->remove_maps_and_stop_waiters
->remove_maps
Here, we skip alloc_path if pp isn't find in pathvec and process is daemon. In
daemon, we should not store path with incomplete information to pathvec. The
pathvec stores all paths in daemon, so it is reasonable keep same with pathvec.
Reported-by: Tianxiong Lu <lutianxiong(a)huawei.com>
Signed-off-by: Lixiaokeng <lixiaokeng(a)huawei.com>
Signed-off-by: Zhiqiang Liu <liuzhiqiang26(a)huawei.com>
---
libmultipath/dmparser.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/libmultipath/dmparser.c b/libmultipath/dmparser.c
index 7da5fbf..3ce6eb3 100644
--- a/libmultipath/dmparser.c
+++ b/libmultipath/dmparser.c
@@ -305,6 +305,16 @@ int disassemble_map(vector pathvec, char *params, struct multipath *mpp,
}
if (!pp) {
+ /* daemon should keep same with pathvec */
+ /* pp is not find in pathvec, skip it */
+ if (is_daemon) {
+ FREE(word);
+ for (k = 0; k < num_paths_args; k++) {
+ p += get_word(p, NULL);
+ }
+ continue;
+ }
+
pp_unfound = 1;
pp = alloc_path();
@@ -317,8 +327,7 @@ int disassemble_map(vector pathvec, char *params, struct multipath *mpp,
strlcpy(pp->wwid, mpp->wwid,
WWID_SIZE);
}
- /* Only call this in multipath client mode */
- if (!is_daemon && store_path(pathvec, pp)) {
+ if (store_path(pathvec, pp)) {
free_path(pp);
goto out1;
}
--
[View Less]
When multipathd del path xxx, multipathd -v2, multipathd add path xxx and multipath -U
dm-x are executed simultaneously, multipath -U dm-x will case coredump.
The reason is that there are two paths with same dev_t in dm_table. The process
is as follows:
multipathd del path xxx(such as sde whose dev_t is 8:64):
cli_del_path
->ev_remove_path
->domap //dm_table in kernel will be reloaded and doesn't contain 8:64.
//Then multipath -v2 is executed, and the dm_table in kernel
…
[View More] //will be reloaded and contains 8:64.
->setup_multipath
->update_multipath_strings
->update_multipath_table
->dm_get_map //get params with 8:64
->disassemble_map //pp1 will be saved mpp->pg
->delete pp1 in pathvec
->clear_ref_from_mpp //pp is cleared in mpp->paths but still saved in
//mpp->pg
->free_paths //pp1 is freed but still exist in mpp->pg and is not null
multipathd add path sde
cli_add_path
->store_pathinfo //alloc pp2 (dev_t is 8:64), and store it to gvecs->pathvec
//pp2 is not equal to pp1
->ev_add_path
->adopt_paths
->update_mpp_paths //pp1 is found in mpp->pg and its dev_t is
//8:64 and dev is not sde (cased by free).
//it will be stored in mpp->paths.
->pp2 is stored to mpp->paths
->setup_map //params with two 8:64
->domap //dm_table is reloaded and contains two 8:64
multipath -U dm-x(sde is one path of dm-x)
main
->check_usable_paths
->dm_get_maps //get params with two 8:64
->disassemble_map //alloc pp3 whose dev_t is 8:64, and pp3 is saved
//twice in mpp->pg
->free_multipath(mpp, FREE_PATHS) //double free pp3
Here, we add that pp1 in mpp->pg is cleared in clear_ref_from_mpp.
Reported-by: Tianxiong Lu <lutianxiong(a)huawei.com>
Signed-off-by: lixiaokeng <lixiaokeng(a)huawei.com>
---
multipathd/main.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/multipathd/main.c b/multipathd/main.c
index e80ab74..a9e7edd 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -818,7 +818,9 @@ static
void clear_ref_from_mpp(struct path * pp, struct vectors * vecs)
{
struct multipath * mpp = NULL;
+ struct pathgroup * pgp;
int i = -1;
+ int j;
mpp = find_mp_by_wwid(vecs->mpvec, pp->wwid);
if(!!mpp){
@@ -826,6 +828,11 @@ void clear_ref_from_mpp(struct path * pp, struct vectors * vecs)
if ((i = find_slot(mpp->paths, (void *)pp)) != -1){
vector_del_slot(mpp->paths, i);
}
+ vector_foreach_slot (mpp->pg, pgp, j) {
+ if ((i = find_slot(pgp->paths, (void *)pp)) != -1){
+ vector_del_slot(pgp->paths, i);
+ }
+ }
}
}
--
[View Less]
Coredump appears when events are executed in a certain order.
1. The sdc and sdd are added as paths of d1. The sdc is added
firstly and d1->wait_for_udev be set to 1. Then sdd is added but
it just adds to pathvec because d1->wait_for_udev is 1.
2. The uev_remove_path(sdc) are executed. Because d1 has one
path, d1 will be flushed.
3. The uev_add_path(sdc) are executed. Now sdc and sdd will
both add to d1 in adopt_paths because sdd->wwid is d1. And
d1->wait_for_udev be set to 1.
4. …
[View More]The uev_remove_path(sdd) are executed. Because d1->wait_for_udev
is 1, domap doesn't executed and the major:minor of sdd(8:64)
still exist in kernel d1. The sdd in pathvec is deleted.
5. The uev_add_path(sdd) are executed. Now the sdd(8:64) is
a path of d2. And new sdd is added to pathvec.
6. The update_multipath_strings(d1) are executed. The d1 find
sdd in pathvec by dev_t(8:64)and sdd will be add to d1. Now
the pointer of sdd is store in paths of d1 and d2.
7. When NIC is down and a new path add to d1, sdd will be freed
in verify_paths (because sdd is actually a path of d2).
8. When d2->hwe is sdd->hwe and setup_map(d2) is called. Core
causes.
The path shouldn't be add to the mpp whose wwid is different
with its. Here we add wwid check in disassemble_map. This bug
has been solved in upstream in update_pathvec_from_dm. It is
a great change, so we fix it with huawei patch. This patch could
be discarded when multipath-tools update.
Signed-off-by:lixiaokeng<lixiaokeng(a)huawei.com>
---
libmultipath/dmparser.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/libmultipath/dmparser.c b/libmultipath/dmparser.c
index 0f370e9..99b57db 100644
--- a/libmultipath/dmparser.c
+++ b/libmultipath/dmparser.c
@@ -342,6 +342,16 @@ int disassemble_map(vector pathvec, char *params, struct multipath *mpp,
goto out1;
}
} else {
+ if (strlen(pp->wwid) && strlen(mpp->wwid) &&
+ strcmp(pp->wwid, mpp->wwid) != 0) {
+ condlog(0, "%s: path wwid is different with that of %s.\n", pp->dev_t, mpp->alias);
+ FREE(word);
+ for (k = 0; k < num_paths_args; k++) {
+ p += get_word(p, NULL);
+ }
+ continue;
+ }
+
if (!strlen(pp->wwid) &&
strlen(mpp->wwid))
strlcpy(pp->wwid, mpp->wwid,
--
[View Less]
When one iscsi device logs in and logs out with the "multipath -r"
executed at the same time, memory leak happens in multipathd
process.
The reason is following. When "multipath -r" is executed, the path
will be free in configure function. Before path_discovery executed,
iscsi device logs out. Then path_discovery will not find any path and
there is no path in the gvecs->pathvec. When map_discovery function
is executed, disassemble_map function will be called. Because
gvecs->pathvec->…
[View More]slot is empty and is_deamon is 1, a path will be
allocated and is not stored in gvecs->pathvec but store in
mpp->pg. But when the mpp is removed and freed by remove_map
function, the path will not be free and can't be find anymore.
The procedure details given as follows,
1."multipath -r" is executed
main
->child
->reconfigure
->configure
->path_discovery //after iscsi logout
->map_discovery
->update_multipath_table
->disassemble_map
->alloc_path
2.then "multipath -r" is executed again
main
->child
->reconfigure
->remove_maps_and_stop_waiters
->remove_maps
Here, we skip alloc_path if pp isn't find in pathvec and process is daemon. In
daemon, we should not store path with incomplete information to pathvec. The
pathvec stores all paths in daemon, so it is reasonable keep same with pathvec.
Reported-by: Tianxiong Lu <lutianxiong(a)huawei.com>
Signed-off-by: Lixiaokeng <lixiaokeng(a)huawei.com>
Signed-off-by: Zhiqiang Liu <liuzhiqiang26(a)huawei.com>
---
libmultipath/dmparser.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/libmultipath/dmparser.c b/libmultipath/dmparser.c
index 7da5fbf..3ce6eb3 100644
--- a/libmultipath/dmparser.c
+++ b/libmultipath/dmparser.c
@@ -305,6 +305,16 @@ int disassemble_map(vector pathvec, char *params, struct multipath *mpp,
}
if (!pp) {
+ /* daemon should keep same with pathvec */
+ /* pp is not find in pathvec, skip it */
+ if (is_daemon) {
+ FREE(word);
+ for (k = 0; k < num_paths_args; k++) {
+ p += get_word(p, NULL);
+ }
+ continue;
+ }
+
pp_unfound = 1;
pp = alloc_path();
@@ -317,8 +327,7 @@ int disassemble_map(vector pathvec, char *params, struct multipath *mpp,
strlcpy(pp->wwid, mpp->wwid,
WWID_SIZE);
}
- /* Only call this in multipath client mode */
- if (!is_daemon && store_path(pathvec, pp)) {
+ if (store_path(pathvec, pp)) {
free_path(pp);
goto out1;
}
--
[View Less]
When multipathd del path xxx, multipathd -v2, multipathd add path xxx and multipath -U
dm-x are executed simultaneously, multipath -U dm-x will case coredump.
The reason is that there are two paths with same dev_t in dm_table. The process
is as follows:
multipathd del path xxx(such as sde whose dev_t is 8:64):
cli_del_path
->ev_remove_path
->domap //dm_table in kernel will be reloaded and doesn't contain 8:64.
//Then multipath -v2 is executed, and the dm_table in kernel
…
[View More] //will be reloaded and contains 8:64.
->setup_multipath
->update_multipath_strings
->update_multipath_table
->dm_get_map //get params with 8:64
->disassemble_map //pp1 will be saved mpp->pg
->delete pp1 in pathvec
->clear_ref_from_mpp //pp is cleared in mpp->paths but still saved in
//mpp->pg
->free_paths //pp1 is freed but still exist in mpp->pg and is not null
multipathd add path sde
cli_add_path
->store_pathinfo //alloc pp2 (dev_t is 8:64), and store it to gvecs->pathvec
//pp2 is not equal to pp1
->ev_add_path
->adopt_paths
->update_mpp_paths //pp1 is found in mpp->pg and its dev_t is
//8:64 and dev is not sde (cased by free).
//it will be stored in mpp->paths.
->pp2 is stored to mpp->paths
->setup_map //params with two 8:64
->domap //dm_table is reloaded and contains two 8:64
multipath -U dm-x(sde is one path of dm-x)
main
->check_usable_paths
->dm_get_maps //get params with two 8:64
->disassemble_map //alloc pp3 whose dev_t is 8:64, and pp3 is saved
//twice in mpp->pg
->free_multipath(mpp, FREE_PATHS) //double free pp3
Here, we add that pp1 in mpp->pg is cleared in clear_ref_from_mpp.
Reported-by: Tianxiong Lu <lutianxiong(a)huawei.com>
Signed-off-by: lixiaokeng <lixiaokeng(a)huawei.com>
---
multipathd/main.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/multipathd/main.c b/multipathd/main.c
index e80ab74..a9e7edd 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -818,7 +818,9 @@ static
void clear_ref_from_mpp(struct path * pp, struct vectors * vecs)
{
struct multipath * mpp = NULL;
+ struct pathgroup * pgp;
int i = -1;
+ int j;
mpp = find_mp_by_wwid(vecs->mpvec, pp->wwid);
if(!!mpp){
@@ -826,6 +828,11 @@ void clear_ref_from_mpp(struct path * pp, struct vectors * vecs)
if ((i = find_slot(mpp->paths, (void *)pp)) != -1){
vector_del_slot(mpp->paths, i);
}
+ vector_foreach_slot (mpp->pg, pgp, j) {
+ if ((i = find_slot(pgp->paths, (void *)pp)) != -1){
+ vector_del_slot(pgp->paths, i);
+ }
+ }
}
}
--
[View Less]