iscsi: add EXITING conns to pg after full_feature_migrate

Commit 41f59559e added code to skip adding EXITING connections
to the new poll group in the full_feature_migrate message
callback.  The problem is that since the connection is in
EXITING state and is not in a poll group, it will never move
to EXITED state, nor get removed from g_active_conns, and
hence will block the iscsi subsystem from being able to
shutdown.

So instead, assert that the connection is not in EXITED
state.  If it is in EXITING state, we will add it to the
poll group, and then when the poll group is next polled,
it will destroy the connection, moving it to EXITED
state and removing it from the g_active_conns STAILQ.

This fix is related to issue #2416.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie8e64c811a5602ba4b28871bc535f5fa49dffc18
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16019
Reviewed-by: Michal Berger <michal.berger@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot

lib/iscsi/conn.c

@@ -1492,11 +1492,13 @@ iscsi_conn_full_feature_migrate(void *arg)
 {
 	struct spdk_iscsi_conn *conn = arg;
 
-	if (conn->state >= ISCSI_CONN_STATE_EXITING) {
+	assert(conn->state != ISCSI_CONN_STATE_EXITED);
-		/* Connection is being exited before this callback is executed. */
+
-		SPDK_DEBUGLOG(iscsi, "Connection is already exited.\n");
+	/* Note: it is possible that connection could have moved to EXITING
-		return;
+	 * state after this message was sent. We will still add it to the
-	}
+	 * poll group in this case.  When the poll group is polled
+	 * again, it will call iscsi_conn_destruct() on it.
+	 */
 
 	if (conn->sess->session_type == SESSION_TYPE_NORMAL) {
 		iscsi_conn_open_luns(conn);