wxWidgets and OpenSSL

Securing connections is always a good idea and OpenSSL provides a number of handy ways to do it. Blocking OpenSSL API is the most easiest to use and it could provide a working solution in no time, however it doesn't work with wxWidgets library so smooth as one may hope. I spent some time to find the reason why my application doesn't work as intended (plain sockets work just fine though) and had to do a little research on that matter.

Regardless of the flags you provide to wxSocketServer it always creates a socket on the new connection with that ioctl call (UnblockAndRegisterWithEventLoop function in include/wx/unix/private/sockunix.h):

ioctl(m_fd, FIONBIO, &trueArg);

Basically, it just makes the socket non-blocking. I don't know is it a bug or feature, but we have it now and have to deal with it. Naturally all blocking OpenSSL API doesn't work and we have to develop the code using non-blocking behaviour. However, it's pretty easy to use if we emulate the blocking pattern in the code.

First, we have to guarantee that the created socket doesn't accept events and works with blocking calls correctly:

auto socket = server->Accept(false);
if (!socket) {
    return;
}
//Notify() cannot be called in thread context. We have to detach from main loop
//before switching thread contexts.
socket->Notify(false);
socket->SetFlags(wxSOCKET_WAITALL|wxSOCKET_BLOCK);

auto thread = new WorkerThread(socket);
if (thread->Create() != wxTHREAD_NO_ERROR) {
    wxLogError(wxT("Can't create thread!"));
} else if (thread->Run() != wxTHREAD_NO_ERROR) {
    wxLogError(wxT("Can't run thread!"));
}

In the thread we process the OpenSSL calls with that helper function:

int NonBlockingSslCall(SSL* ssl, wxSocketBase* socket,
                       std::function<int ()> func) {
    int ret;
    for(;;) {
        ret = func();
        if (ret < 0 && socket->IsOk()) {
            switch (SSL_get_error(ssl, ret)) {
                case SSL_ERROR_WANT_READ:
                    socket->WaitForRead();
                    break;
                case SSL_ERROR_WANT_WRITE:
                    socket->WaitForWrite();
                    break;
                default:
                    throw SslException("Fatal error occurred", ssl, ret);
            }
            continue;
        }
        break;
    }
    return ret;
}

In few words this function do the OpenSSL call (see the func variable) and check for SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE error code. If it happens, it blocks the program waiting for the corresponding data.

Here comes C++11 handy features - and we use lambda to do the required "blocking" OpenSSL call:

int r = NonBlockingSslCall(ssl, socket,
    [ssl] () { return SSL_accept(ssl); });
if (r <= 0) {
    throw SslException("The SSL accept operation failed", ssl, r);
}

One more notice here - lambda doesn't work with moveable types, so you have to use the plain pointer as a catching parameter:

std::unique_ptr<BIO, decltype(&BIO_free)> io(BIO_new(BIO_f_buffer()), BIO_free);
...
BIO* io_ptr = io.get();
r = NonBlockingSslCall(ssl, socket,
        [io_ptr, buf_ptr, out_ptr] () {
            int sz = BIO_gets(io_ptr, buf_ptr, BUFSIZZ - 1);
            if (sz > 0) {
                out_ptr->Write(buf_ptr, sz);
            }
            return sz;
        }
    );

The examples above don't provide full listings of the code, but I hope the idea is clear. Surely, there are more elegant (but I guess more verbose) solutions like by using wxWidgets events, but the method above works for me without any problems. I hope that will help you too on the rough road of securing your networking software.