Websocket 传输可靠性(重新连接期间 Socket.io 数据丢失)

二手货

NodeJS Socket.io

问题

假设有两个用户 U1U2,通过 Socket.io 连接到一个应用程序:

  1. U1 完全失去互联网连接(例如,关闭互联网)
  2. U2 U1发送消息。
  3. U1 还没有收到消息,因为互联网已经关闭
  4. Server 通过心跳超时检测 U1断开连接
  5. U1 重新连接到 socket.io
  6. U1 从来没有收到来自 U2的消息-我猜它在第4步中丢失了。

可能的解释

我想我知道为什么会这样:

  • 第4步 服务器杀死套接字实例和到 U1的消息队列
  • 此外,在步骤5 U1服务器上创建新的连接(它不会被重用) ,因此即使消息仍然在队列中,以前的连接也会丢失。

需要帮助

如何防止这类数据丢失?我必须使用心跳,因为我不会让人们永远挂在应用程序上。此外,我还必须提供重新连接的可能性,因为当我部署新版本的应用程序时,我希望零停机时间。

附言。我称之为“消息”的东西不仅是可以存储在数据库中的文本消息,还是有价值的系统消息,必须保证传递,否则用户界面会搞砸。

谢谢!


加1

我已经有一个用户账户系统了。而且,我的应用程序已经很复杂了。添加离线/在线状态不会有任何帮助,因为我已经有了这类东西。问题不一样。

看看第二步。在这一步我们技术上 无法确定 U1是否离线,他只是失去连接让说2秒钟,可能是因为互联网不好。所以 U2给他发了一条信息,但是 U1没有收到,因为他还没有上网(第三步)。需要步骤4来检测脱机用户,比方说,超时60秒。最终在另一个10秒钟的互联网连接为 U1是起来了,他重新连接到 socket.io。但是来自 U2的消息在太空中丢失了,因为服务器上的 U1被超时断开了连接。

这就是问题所在,我不想百分百交货。


解决方案

  1. 在随机 emitID 标识的{}用户中收集发出(发出名称和数据)
  2. 在客户端确认发出(使用 emitID 将发出发回服务器)
  3. 如果确认-从 emitID 标识的{}中删除对象
  4. 如果用户重新连接-检查该用户的{} ,并循环通过它对{}中的每个对象执行步骤1
  5. 当断开连接或/和已连接时,如果需要,为用户刷新{}
// Server
const pendingEmits = {};


socket.on('reconnection', () => resendAllPendingLimits);
socket.on('confirm', (emitID) => { delete(pendingEmits[emitID]); });


// Client
socket.on('something', () => {
socket.emit('confirm', emitID);
});

解决方案2(某种程度上)

增补2020年2月1日。

虽然这不是 Websockets 的真正解决方案,但有些人可能仍然觉得它很方便。我们从 Websockets 迁移到了 SSE + Ajax。SSE 允许您从客户端进行连接,以保持持久的 TCP 连接,并实时从服务器接收消息。要将消息从客户机发送到服务器,只需使用 Ajax。存在延迟和开销等缺点,但 SSE 保证可靠性,因为它是 TCP 连接。

因为我们使用 Express,所以我们对 SSE https://github.com/dpskvn/express-sse使用这个库,但是您可以选择一个适合您的库。

在 IE 和大多数 Edge 版本中不支持 SSE,因此需要一个 polyfill: https://github.com/Yaffle/EventSource

40885 次浏览

It is seem that you already have user account system. You know which account is online/offline, you you can handle connect/disconnect event:

So the solution is, add online/offline and offline messages on database for each user:

chatApp.onLogin(function (user) {
user.readOfflineMessage(function (msgs) {
user.sendOfflineMessage(msgs, function (err) {
if (!err) user.clearOfflineMessage();
});
})
});


chatApp.onMessage(function (fromUser, toUser, msg) {
if (user.isOnline()) {
toUser.sendMessage(msg, function (err) {
// alert CAN NOT SEND, RETRY?
});
} else {
toUser.addToOfflineQueue(msg);
}
})

Look here: Handle browser reload socket.io.

I think you could use solution which I came up with. If you modify it properly, it should work as you want.

What I think you want is to have a reusable socket for each user, something like:

Client:

socket.on("msg", function(){
socket.send("msg-conf");
});

Server:

// Add this socket property to all users, with your existing user system
user.socket = {
messages:[],
io:null
}
user.send = function(msg){ // Call this method to send a message
if(this.socket.io){ // this.io will be set to null when dissconnected
// Wait For Confirmation that message was sent.
var hasconf = false;
this.socket.io.on("msg-conf", function(data){
// Expect the client to emit "msg-conf"
hasconf = true;
});
// send the message
this.socket.io.send("msg", msg); // if connected, call socket.io's send method
setTimeout(function(){
if(!hasconf){
this.socket = null; // If the client did not respond, mark them as offline.
this.socket.messages.push(msg); // Add it to the queue
}
}, 60 * 1000); // Make sure this is the same as your timeout.


} else {
this.socket.messages.push(msg); // Otherwise, it's offline. Add it to the message queue
}
}
user.flush = function(){ // Call this when user comes back online
for(var msg in this.socket.messages){ // For every message in the queue, send it.
this.send(msg);
}
}
// Make Sure this runs whenever the user gets logged in/comes online
user.onconnect = function(socket){
this.socket.io = socket; // Set the socket.io socket
this.flush(); // Send all messages that are waiting
}
// Make sure this is called when the user disconnects/logs out
user.disconnect = function(){
self.socket.io = null; // Set the socket to null, so any messages are queued not send.
}

Then the socket queue is preserved between disconnects.

Make sure it saves each users socket property to the database and make the methods part of your user prototype. The database does not matter, just save it however you have been saving your users.

This will avoid the problem mentioned in Additon 1 by requiring a confirmation from the client before marking the message as sent. If you really wanted to, you could give each message an id and have the client send the message id to msg-conf, then check it.

In this example, user is the template user that all users are copied from, or like the user prototype.

Note: This has not been tested.

Others have hinted at this in other answers and comments, but the root problem is that Socket.IO is just a delivery mechanism, and you cannot depend on it alone for reliable delivery. The only person who knows for sure that a message has been successfully delivered to the client is the client itself. For this kind of system, I would recommend making the following assertions:

  1. Messages aren't sent directly to clients; instead, they get sent to the server and stored in some kind of data store.
  2. Clients are responsible for asking "what did I miss" when they reconnect, and will query the stored messages in the data store to update their state.
  3. If a message is sent to the server while the recipient client is connected, that message will be sent in real time to the client.

Of course, depending on your application's needs, you can tune pieces of this--for example, you can use, say, a Redis list or sorted set for the messages, and clear them out if you know for a fact a client is up to date.


Here are a couple of examples:

Happy path:

  • U1 and U2 are both connected to the system.
  • U2 sends a message to the server that U1 should receive.
  • The server stores the message in some kind of persistent store, marking it for U1 with some kind of timestamp or sequential ID.
  • The server sends the message to U1 via Socket.IO.
  • U1's client confirms (perhaps via a Socket.IO callback) that it received the message.
  • The server deletes the persisted message from the data store.

Offline path:

  • U1 looses internet connectivity.
  • U2 sends a message to the server that U1 should receive.
  • The server stores the message in some kind of persistent store, marking it for U1 with some kind of timestamp or sequential ID.
  • The server sends the message to U1 via Socket.IO.
  • U1's client does not confirm receipt, because they are offline.
  • Perhaps U2 sends U1 a few more messages; they all get stored in the data store in the same fashion.
  • When U1 reconnects, it asks the server "The last message I saw was X / I have state X, what did I miss."
  • The server sends U1 all the messages it missed from the data store based on U1's request
  • U1's client confirms receipt and the server removes those messages from the data store.

If you absolutely want guaranteed delivery, then it's important to design your system in such a way that being connected doesn't actually matter, and that realtime delivery is simply a bonus; this almost always involves a data store of some kind. As user568109 mentioned in a comment, there are messaging systems that abstract away the storage and delivery of said messages, and it may be worth looking into such a prebuilt solution. (You will likely still have to write the Socket.IO integration yourself.)

If you're not interested in storing the messages in the database, you may be able to get away with storing them in a local array; the server tries to send U1 the message, and stores it in a list of "pending messages" until U1's client confirms that it received it. If the client is offline, then when it comes back it can tell the server "Hey I was disconnected, please send me anything I missed" and the server can iterate through those messages.

Luckily, Socket.IO provides a mechanism that allows a client to "respond" to a message that looks like native JS callbacks. Here is some pseudocode:

// server
pendingMessagesForSocket = [];


function sendMessage(message) {
pendingMessagesForSocket.push(message);
socket.emit('message', message, function() {
pendingMessagesForSocket.remove(message);
}
};


socket.on('reconnection', function(lastKnownMessage) {
// you may want to make sure you resend them in order, or one at a time, etc.
for (message in pendingMessagesForSocket since lastKnownMessage) {
socket.emit('message', message, function() {
pendingMessagesForSocket.remove(message);
}
}
});


// client
socket.on('connection', function() {
if (previouslyConnected) {
socket.emit('reconnection', lastKnownMessage);
} else {
// first connection; any further connections means we disconnected
previouslyConnected = true;
}
});


socket.on('message', function(data, callback) {
// Do something with `data`
lastKnownMessage = data;
callback(); // confirm we received the message
});

This is quite similar to the last suggestion, simply without a persistent data store.


You may also be interested in the concept of event sourcing.

Been looking at this stuff latterly and think different path might be better.

Try looking at Azure Service bus, ques and topic take care of the off line states. The message wait for user to come back and then they get the message.

Is a cost to run a queue but its like $0.05 per million operations for a basic queue so cost of dev would be more from hours work need to write a queuing system. https://azure.microsoft.com/en-us/pricing/details/service-bus/

And azure bus has libraries and examples for PHP, C#, Xarmin, Anjular, Java Script etc.

So server send message and does not need to worry about tracking them. Client can use message to send back also as means can handle load balancing if needed.

Try this emit chat list

io.on('connect', onConnect);


function onConnect(socket){


// sending to the client
socket.emit('hello', 'can you hear me?', 1, 2, 'abc');


// sending to all clients except sender
socket.broadcast.emit('broadcast', 'hello friends!');


// sending to all clients in 'game' room except sender
socket.to('game').emit('nice game', "let's play a game");


// sending to all clients in 'game1' and/or in 'game2' room, except sender
socket.to('game1').to('game2').emit('nice game', "let's play a game (too)");


// sending to all clients in 'game' room, including sender
io.in('game').emit('big-announcement', 'the game will start soon');


// sending to all clients in namespace 'myNamespace', including sender
io.of('myNamespace').emit('bigger-announcement', 'the tournament will start soon');


// sending to individual socketid (private message)
socket.to(<socketid>).emit('hey', 'I just met you');


// sending with acknowledgement
socket.emit('question', 'do you think so?', function (answer) {});


// sending without compression
socket.compress(false).emit('uncompressed', "that's rough");


// sending a message that might be dropped if the client is not ready to receive messages
socket.volatile.emit('maybe', 'do you really need it?');


// sending to all clients on this node (when using multiple nodes)
io.local.emit('hi', 'my lovely babies');


};

Michelle's answer is pretty much on point, but there are a few other important things to consider. The main question to ask yourself is: "Is there a difference between a user and a socket in my app?" Another way to ask that is "Can each logged in user have more than 1 socket connection at one time?"

In the web world it is probably always a possibility that a single user has multiple socket connections, unless you have specifically put something in place that prevents this. The simplest example of this is if a user has two tabs of the same page open. In these cases you don't care about sending a message/event to the human user just once... you need to send it to each socket instance for that user so that each tab can run it's callbacks to update the ui state. Maybe this isn't a concern for certain applications, but my gut says it would be for most. If this is a concern for you, read on....

To solve this (assuming you are using a database as your persistent storage) you would need 3 tables.

  1. users - which is a 1 to 1 with real people
  2. clients - which represents a "tab" that could have a single connection to a socket server. (any 'user' may have multiple)
  3. messages - a message that needs sent to a client (not a message that needs sent to a user or to a socket)

The users table is optional if your app doesn't require it, but the OP said they have one.

The other thing that needs properly defined is "what is a socket connection?", "When is a socket connection created?", "when is a socket connection reused?". Michelle's psudocode makes it seem like a socket connection can be reused. With Socket.IO, they CANNOT be reused. I've seen be the source of a lot of confusion. There are real life scenarios where Michelle's example does make sense. But I have to imagine those scenarios are rare. What really happens is when a socket connection is lost, that connection, ID, etc will never be reused. So any messages marked for that socket specifically will never be delivered to anyone because when the client who had originally connected, reconnects, they get a completely brand new connection and new ID. This means it's up to you to do something to track clients (rather than sockets or users) across multiple socket connections.

So for a web based example here would be the set of steps I'd recommend:

  • When a user loads a client (typically a single webpage) that has the potential for creating a socket connection, add a row to the clients database which is linked to their user ID.
  • When the user actually does connect to the socket server, pass the client ID to the server with the connection request.
  • The server should validate the user is allowed to connect and the client row in the clients table is available for connection and allow/deny accordingly.
  • Update the client row with the socket ID generated by Socket.IO.
  • Send any items in the messages table connected to the client ID. There wouldn't be any on initial connection, but if this was from the client trying to reconnect, there may be some.
  • Any time a message needs to be sent to that socket, add a row in the messages table which is linked to the client ID you generated (not the socket ID).
  • Attempt to emit the message and listen for the client with the acknowledgement.
  • When you get the acknowledgement, delete that item from the messages table.
  • You may wish to create some logic on the client side that discards duplicate messages sent from the server since this is technically a possibility as some have pointed out.
  • Then when a client disconnects from the socket server (purposefully or via error), DO NOT delete the client row, just clear out the socket ID at most. This is because that same client could try to reconnect.
  • When a client tries to reconnect, send the same client ID it sent with the original connection attempt. The server will view this just like an initial connection.
  • When the client is destroyed (user closes the tab or navigates away), this is when you delete the client row and all messages for this client. This step may be a bit tricky.

Because the last step is tricky (at least it used to be, I haven't done anything like that in a long time), and because there are cases like power loss where the client will disconnect without cleaning up the client row and never tries to reconnect with that same client row - you probably want to have something that runs periodically to cleanup any stale client and message rows. Or, you can just permanently store all clients and messages forever and just mark their state appropriately.

So just to be clear, in cases where one user has two tabs open, you will be adding two identical message to the messages table each marked for a different client because your server needs to know if each client received them, not just each user.

As already written in another answer, I also believe you should look at the realtime as a bonus : the system should be able to work also with no realtime.

I’m developing an enterprise chat for a large company (ios, android, web frontend and .net core + postGres backend) and after having developed a way for the websocket to re-establish connection (through a socket uuid) and get undelivered messages (stored in a queue) I understood there was a better solution: resync via rest API.

Basically I ended up by using websocket just for realtime, with an integer tag on each realtime message (user online, typers, chat message and so on) for monitoring lost messages.

When the client gets an id which is not monolithic (+1) then it understands it is out of sync so it drops all the socket messages and asks a resync of all its observers through REST api.

This way we can handle many variations in the state of the application during the offline period without having to parse tons of websocket messages in a row on reconnection and we are sure to be synced (because the last sync date is set just by the REST api, not from the socket).

The only tricky part is monitoring for realtime messages from the moment you call REST api to the moment the server replies because what is read from the db takes time to get back to the client and in the meanwhile variations could happen so they need to be cached and took into account.

We are going into production in a couple of months, I hope to get back sleeping by then :)