Client initiated graceful close causes 995 Windows error (connection forcibly closed)

[stavikpetr]

Hello. I have a issue related to the closing of a websocket connection gracefully from the client side while using boost::beast client, because each close attempt results in 995 windows socket error, which means that the connection was forcibly closed on the remote side.

boost version: 1.74
uwebsockets version: 18.17.0 (I also tried the newest version)

To give you a little bit more details, the client is asynchronous and it uses the powerfull boost::asio library in combination with already mentioned boost::beast library. Basically, in the client, I have one thread that is processing all of the asynchronous events in the boost::asio::io_context::run method. The io_context has always one read callback enqueued, that is processing the messages from the server, and its main part looks like this:

void Client::onRead(boost::beast::error_code ec, size_t bytesRead)
{
    if (ec.value() != 0) {
        if (ec == boost::beast::websocket::error::closed) {
            // graceful close, we can return
            return;
        }
        else {
            // ungraceful close, throw exception
        }
    }
    // ...  other code related to the message processing and enqueuing another callback
}

Receiving the messages and sending them works perfectly fine. However, when another thread intiates the shutdown by performing boost::beast::websocket::stream<boost::asio::ip::tcp::socket>>::async_close, I should receive, according to the documentation, the error code boost::beast::websocket::error::closed which indicates clear shutdown. However, instead of that, I always receive the 995 error code. I checked some other examples on the client, and I dont think the issue is in it.

So, I also checked the server, and after some debugging, it seems that the issue is in this part of code, in file WebSocketContext.h:

/* If uncorking was successful and we are in shutdown state then send TCP FIN */
if (asyncSocket->getBufferedAmount() == 0) {
    /* We can now be in shutdown state */
    if (webSocketData->isShuttingDown) {
          /* Shutting down a closed socket is handled by uSockets and just fine */
          asyncSocket->shutdown();
    }
}

More precisely, after the call to asyncSocket->shutdown(), the client receives the aforementioned 995 error and nothing else.

Can anybody please try to help, or at least point me in the right direction? From my limited understanding of the code, it seems that the socket is closed before the shutdown is called, which causes this error. This would be supported by the fact, that when the graceful close is initiated from the server side (after some inactivity), everything is fine, because the socket is not corked and this part of code in the end() function sends the shutdown right after sending the close frame (file WebSocket.h)

WebSocket<SSL, true> *webSocket = (WebSocket<SSL, true> *) this;
if (!webSocket->isCorked()) {
    if (ok) {
         /* If we are not corked, and we just sent off everything, we need to FIN right here.
         * In all other cases, we need to fin either if uncork was successful, or when drainage is complete. */
         webSocket->shutdown(); //<-- this is not called, because the socket is corked
     }
}

Any help is highly appreciated, because currently I am kinda lost on what to do with this. I can provide more details if necessary. And sorry for a long post.

EDIT:
So, I did some additional investigation and I tried to create basically a minimalist reproducible example so it can clearly ilustrate the issue I have in mind. Below, you can see a very simple client and server and I still got the same error even in this case. What is the issue? Any ideas? It just seems that I can not read anything from the websocket client after I send the close event.

Client code

#include <iostream>
#include <string>
#include <memory>
#include <thread>
#include <chrono>
#include <atomic>

#include <boost/beast/websocket.hpp>
#include <boost/asio/io_service.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/beast/core.hpp>

struct ConnectionContext {
	boost::asio::io_context ios;
	boost::beast::websocket::stream<boost::asio::ip::tcp::socket> ws;
	boost::beast::flat_buffer readBuffer;
	
	ConnectionContext() :ws(ios)
	{
	}
};

std::unique_ptr<ConnectionContext> createCC(const std::string& ip, int port)
{
	auto cc = std::make_unique<ConnectionContext>();

	boost::asio::ip::tcp::resolver resolver(cc->ios);
	boost::asio::connect(cc->ws.next_layer(), resolver.resolve(ip, std::to_string(port)));
	cc->ws.handshake(ip + ":" + std::to_string(port), "/");

	return std::move(cc);
}

void onRead(ConnectionContext* cc, boost::beast::error_code ec, size_t bytesRead)
{
	if (ec.value() != 0) {
		if (ec == boost::beast::websocket::error::closed) {
			std::cout << "<<< onRead() - graceful close received" << std::endl;
			return;
		}
		else {
			std::cout << "<<< onRead() - exception during read ... code: " << std::to_string(ec.value()) << std::endl;
			return;
		}
	}

	auto data = std::string((char*)cc->readBuffer.data().data(), cc->readBuffer.data().size());

	std::cout << "<<< onRead() - message received: " << data << std::endl;

	cc->readBuffer.clear();

	// enqueue another async read
	cc->ws.async_read(cc->readBuffer, boost::beast::bind_front_handler(&onRead, cc));
}

void runAsyncClient(ConnectionContext* cc)
{
	// enqueue first async read
	cc->ws.async_read(cc->readBuffer, boost::beast::bind_front_handler(&onRead, cc));

	// run the ios loop
	cc->ios.run();
}

void onWrite(const std::string& msg, boost::beast::error_code ec, size_t bytesWritten)
{
	if (ec.value() != 0) {
		std::cout << "--- message written with error" << std::endl;
	}
	else {
		std::cout << ">>> message written: " << msg << std::endl;
	}
}

void onClose(boost::beast::error_code ec)
{
	if (ec.value() != 0) {
		std::cout << "--- close send failed" << std::endl;
	}
	else {
		std::cout << ">>> close send successfull" << std::endl;
	}
}

int main()
{
	std::string ip = "127.0.0.1";
	auto port = 8000;

	// create connection context - basically connect the client
	std::cout << "--- Creating Connection context " << std::endl;
	auto cc = createCC(ip, port);
	std::cout << "--- Connection context created - client is connected" << std::endl;

	// run the "client connection" in another thread
	std::thread t([&]() {
		runAsyncClient(cc.get());
	});

	// do some dummy communication
	std::this_thread::sleep_for(std::chrono::seconds(1));
	std::cout << "--- dummy subscribe" << std::endl;
	std::string msg1("subscribe topic X");
	cc->ws.async_write(boost::asio::buffer(msg1), boost::beast::bind_front_handler(&onWrite, msg1));

	std::this_thread::sleep_for(std::chrono::seconds(5));
	std::cout << "--- dummy unsubscribe" << std::endl;
	std::string msg2("unsubscribe topic X");
	cc->ws.async_write(boost::asio::buffer(msg2), boost::beast::bind_front_handler(&onWrite, msg2));

	// close the connection
	std::this_thread::sleep_for(std::chrono::seconds(5));
	std::cout << "--- sending close" << std::endl;
	cc->ws.async_close(boost::beast::websocket::close_code::normal, boost::beast::bind_front_handler(&onClose));

	t.join();
}

Server code

#include <iostream>
#include <string>
struct IUnknown;

#define UWS_NO_ZLIB
#define UWS_NO_SSL
#define LIBUS_NO_SSL
#define LIBUS_NO_LIBUV
#undef ERROR
#include <uwebsockets/App.h>

class ClientContext {
};

void processMessage(auto* ws, std::string_view msg)
{
	std::cout << "<<< received message: " << msg << std::endl;

	auto response = std::string("echo :") + std::string(msg);
	ws->send(response, uWS::TEXT);

	std::cout << ">>> response: " << response << std::endl;
}

uWS::App::WebSocketBehavior webSocketBehavior()
{
	uWS::App::WebSocketBehavior config;
	config.compression = uWS::DISABLED;
	config.maxPayloadLength = 100 * 1024;

	config.open = [](auto* ws)
	{
		std::cout << "--- new client connected" << std::endl;
	};

	config.message = [](auto* ws, auto message, auto opCode) {
		processMessage(ws, message);
	};

	config.close = [](auto* ws, auto code, auto message) {
		std::cout << "--- performing client close callback" << std::endl;
	};

	return std::move(config);
}

int main()
{
	uWS::App app;
	app.ws<ClientContext>("/*", std::move(webSocketBehavior()))
		.listen(8000, [&](us_listen_socket_t* socket) {
		if (socket) {
			std::cout << "--- Server is listening on port 8000" << std::endl;
		}
		else {
			std::cout << "--- Server failed to listen" << std::endl;
		}
	})
	.run();
	
}