QT/WebKit::Adding New Protocols to QtWebKit

The QtWebKit module presents many ways to integrate the worlds of native desktop and mobile applications and the Web, making it possible for developers to extend and combine features found in Qt and WebKit to create new ones. In this article, we examine the use of Qt's network access API with WebKit and show how to turn QWebView into a simple FTP client.

In Qt Quarterly 26 article, , we extended Qt's WebKit integration by showing how to add custom widgets to Web pages. In the article, we used to ask for content for display in a widget, and we obtained the data returned by the server by reading from the corresponding .

Qt's network access API is a technology that aims to replace much, but not all, of the functionality provided by the and classes. We used in the Qt Quarterly 23 article, . Although the network access API is a Qt-specific technology, the module integrates this Qt technology with WebKit to enable customization of the browser engine by Qt application developers. It also means that we can control how the browser engine obtains and renders content.

Since and are designed to provide a reusable abstraction for network operations, it seems obvious to use these classes to add FTP support to browsers written using QtWebKit. To do this, we first need to examine the network access classes before we see how the QtWebKit module uses them to manage network operations.

Network Access

The central class in Qt's network access API is . This class performs the work of dispatching requests to remote servers and handling incoming replies. Applications typically construct an instance of this class and use it for all high level network communication.

Applications create objects, each of them specifying a URL where the request is to be sent and containing meta-data that will be understood by the server. Each request is dispatched by passing it to a function in the network manager — there are different functions corresponding to different kinds of operations, such as , and . Each of these functions returns a object which is used to obtain the content sent in the reply, as well as any meta-data that describes it.

The QtWebKit module provides the class which represents the content displayed in a widget. Behind the scenes, this class uses a default network access manager to handle network communication. This default manager works perfectly well for fetching content over HTTP from http:// URLs, but only supports fetching of files over FTP when using ftp:// URLs.

Fortunately, QWebPage provides the function that allows the default manager to be replaced with one with more features. This lets us add improved support for FTP quite easily if we can write a new manager that supports ftp:// URLs.

The process of replacing the manager and using a new one with an existing object can be broken up into three steps:

1. Creating a new subclass.
2. Creating a new subclass to deal with the FTP protocol.
3. Setting the new manager on the .

Additionally, to provide a reasonable user experience, we should also handle content that the browser engine cannot display. To do this, we create a custom Downloader object. We will briefly return to this topic later.

Creating a New Network Manager

Replacing an existing network manager for a is conceptually simple: we subclass and reimplement its function to check for URLs with the ftp scheme. However, we want to ensure that the manager uses any existing cache and proxy settings that may have been set up for the existing manager used by the .

To keep the existing proxy and cache, we give our network manager a constructor that accepts the old manager as an argument. In the constructor, we reuse the settings from the old manager.

NetworkAccessManager::NetworkAccessManager( *manager,  *parent)
    : (parent)
{
    (manager->());
    (manager->());
    (manager->());
    (manager->());
}

The createRequest() function is used to create and dispatch requests to remote servers for each of the different kinds of operation that the API presents to the developer. Since we are only interested in performing simple fetches of resources using the ftp scheme, we filter out other schemes and other kinds of operation, delegating the task of handling these to the default implementation.

 *NetworkAccessManager::createRequest(
     operation, const  &request,
    QIODevice *device)
{
    if (request.().() != "ftp")
        return (operation, request, device);

    if (operation == )
        return new FtpReply(request.());
    else
        return (operation, request, device);
}

Here, we construct and return an instance of the FtpReply class. This class performs most of the work of handling the FTP protocol.

Creating a Custom Reply

The network access API is designed to be simple to use: we set up a request, dispatch it using the network manager, and obtain a object. If we are not interested in the reply's meta-data, we can simply read the data using its function because is a QIODevice subclass.

In order to keep the API so simple, however, we need to perform some work behind the scenes. In this case, that means that we must perform a series of communications with the FTP server. Fortunately, we can use the existing implementation provided by to perform the low level work.

In the FtpReply class, we need to reimplement four functions in the QIODevice API to ensure that it will work correctly. These functions – abort(), bytesAvailable(), isSequential(), readData() – rely on the rest of the implementation to fill a with data and use an integer offset to track how much has been read from the device by the browser.

class FtpReply : public 
{
    

public:
    FtpReply(const  &url);
    void abort();
    qint64 bytesAvailable() const;
    bool isSequential() const;

protected:
    qint64 readData(char *data, qint64 maxSize);

private slots:
    void processCommand(int command, bool error);
    void processListInfo(const  &urlInfo);
    void processData();

private:
    void setContent();
    void setListContent();

     *ftp;
    <> items;
     content;
    qint64 offset;
     units;
};    

The processCommand(), processListInfo and processData() slots handle interaction with the FTP server. The private setContent() and setListContent() functions are used to add meta-data to the reply and compose HTML for the browser to display.

Two of the private variables hold information about the data obtained from the FTP server: items is updated to contain information about each file found at a given URL, and content contains the raw data obtained from the server. The offset variable is used to track how much data has been read by the browser from the reply.

In the constructor, we construct a object and connect the signals and slots that form the basis of the interaction with the FTP server. The high level communication is reported by the signal. New data from the server is reported by the signal. Individual items in an FTP directory listing are reported by the signal.

FtpReply::FtpReply(const  &url)
    : ()
{
    ftp = new (this);
    (ftp, SIGNAL(()), this, SLOT(processListInfo()));
    (ftp, SIGNAL(()), this, SLOT(processData()));
    (ftp, SIGNAL((int, bool)), this, SLOT(processCommand(int, bool)));

    offset = 0;
    units = () << "bytes" << "K" << "M" << "G" << "Ti" << "Pi"
                          << "Ei" << "Zi" << "Yi";

    (url);
    ftp->(url.());
}

We also initialize the offset into the data that represents the number of bytes that the browser has read from the reply. Additionally, we define a list of units for use with the setListContent() function. The last two tasks performed in the constructor are to set the URL of the reply so that the browser can tell where it came from, and to connect to the FTP server.

Fetching Data from the Server

All communication with the server is handled by the processCommand() slot, which acts on responses from the server and tells us when a command we have issued has completed. This slot performs the task of logging in to the server when connection has occurred (the command has completed), asking for a list of files when logged in ( has completed), preparing a page with a listing when all file information has been received ( has completed), and setting the current content for the reply when data has been fetched from the server ( has completed).

void FtpReply::processCommand(int, bool err)
{
    if (err) {
        (, "Unknown command");
        emit ();
        return;
    }

    switch (ftp->currentCommand()) {
    case :
        ftp->();
        break;

    case :
        ftp->(().());
        break;

    case :
        if (items.() == 1)
            ftp->(().());
        else
            setListContent();
        break;

    case :
        setContent();

    default:
        ;
    }
}

The result of the command is handled by looking at the number of items obtained from the server. The items themselves are recorded by the processListInfo() slot. When a command is complete, we can count the number of items received and determine whether or not we should create a file listing, or try to fetch the file instead by invoking a command.

void FtpReply::processListInfo(const  &urlInfo)
{
    items.(urlInfo);
}

Since the reply will only be used once, we can simply append items to a list and never bother to clear it.

The processData() slot simply appends data obtained from the FTP server to the containing the content to be supplied to the browser.

void FtpReply::processData()
{
    content += ftp->();
}

Data is appended to the content array until the connection to the FTP server is closed, either by the reply or by the server itself. One of the ways in which this happens is when a command completes. At this point, the setContent() function is called from within the processCommand() function.

void FtpReply::setContent()
{
    open(ReadOnly | Unbuffered);
    (, (content.()));
    emit ();
    emit ();
    ftp->();
}

Here, we prepare the reply for use by the browser by opening it for unbuffered reading and setting the header that reports the amount of data held by the reply. We emit signals that indicate that the network operation has finished and that it has data to be read. Since we are no longer interested in the FTP server, we close the connection to it.

Preparing Content for the Reader

Another way in which the reply closes the connection to the server is when the setListContent() function is called from the processCommand() function. Most of the implementation of this function involves transforming the information about the items held in the reply's private items variable to HTML.

void FtpReply::setListContent()
{
     u = ();
    if (!u.().("/"))
        u.(u.() + "/");

     base_url = ().();
     base_path = u.();

    open(ReadOnly | Unbuffered);
     content(
        "\n"
        "\n"
        "  \n"
        "  \n"
        "\n\n"
        "\n"
        "
Listing for " + base_path + "
\n\n"
        "\n"
"\n");

     parent = u.((".."));

if (parent.(u))

        content += ("\n");

    int i = 0;
    foreach (const  &item, items) {

         child = u.((item.name()));

if (i == 0)
            content += ("");
else
            content += ("");

        content += ("");

        qint64 size = item.();
        int unit = 0;
while (size) {
            qint64 new_size = size/1024;
if (new_size && unit < units.()) {
                size = new_size;
                unit += 1;
            } else
break;
        }

if (item.())
            content += ("\n");
else
            content += ("\n");

        i = 1 - i;
    }

    content += ("
        
Name
Size
    
 + parent.() + "\">"
            "Parent directory
        
        
 + child.toString() + "\">"
                           + (item.()) + "
                                            
" + (size) + " "
                               + units[unit] + "
        
\n"
                       "
\n"
                       "\n");

    this->content = content.();

    (, ("text/html; charset=UTF-8"));
    (, (this->content.()));
    emit ();
    emit ();
    ftp->();
}

Once the HTML description of the files has been composed in a , we convert it to a UTF-8 encoded set of bytes which we store in the reply's private content variable. In this case, the holds HTML instead of file data. We set the reply's headers to indicate that it contains UTF-8 encoded HTML with a certain length, and we emit the and signals to let the browser know that it can start reading the content.

Supplying Data to the Browser

We reimplement four QIODevice functions to provide basic read-only behavior, simply supplying the data held in the content array.

We do not support aborting of the reply, so our abort() implementation is empty.

void FtpReply::abort()
{
}

bool FtpReply::isSequential() const
{
    return true;
}

Similarly, we do not support random access reading, so isSequential() is reimplemented to always return true.

The bytesAvailable() function returns the total number of bytes held by the reply minus the value of offset, which is the number of bytes we have already supplied to the reader.

qint64 FtpReply::bytesAvailable() const
{
    return content.() - offset;
}

qint64 FtpReply::readData(char *data, qint64 maxSize)
{
    if (offset < content.()) {
        qint64 number = (maxSize, content.() - offset);
        memcpy(data, content.() + offset, number);
        offset += number;
        return number;
    } else
        return -1;
}

The readData() reimplementation tries to return as much data to the reader as it will allow, copying bytes directly to the appropriate location in memory. The offset variable is updated to keep track of how many bytes have been read.

Enabling the Protocol

Now that we have an FTP-enabled network manager and a reply that can handle communication with FTP servers, we can now enable the manager for a given . For the example included with this article, we derive the FtpView class from and configure its behavior in its constructor.

As we mentioned earlier, we pass the original network manager to the newly-created manager and pass the new manager to the belonging to the browser. This enables our network manager for the content it displays.

FtpView::FtpView()
{
     *oldManager = ()->();
    NetworkAccessManager *newManager = new NetworkAccessManager(oldManager, this);
    ()->(newManager);

    ()->(true);
    downloader = new Downloader(this, newManager);

    connect((), SIGNAL(( *)),
            downloader, SLOT(saveFile( *)));
    connect((), SIGNAL((const  &)),
            downloader, SLOT(startDownload(const  &)));

    connect(this, SIGNAL((const  &)),
            this, SLOT(updateWindowTitle(const  &)));
}

We also go to some effort to handle content that WebKit does not natively support, using a Downloader helper class to manage this and files that the user downloads via the browser's Save Link... context menu entry.

In the example's main() function, we perform the usual steps to initialize our Qt application. We choose an appropriate starting URL for the FtpView widget to open before running the application's event loop.

int main(int argc, char *argv[])
{
     app(argc, argv);

    FtpView view;
    view.(("ftp://ftp.qt.nokia.com"));
    view.();

    return app.();
}

Summary

As we have seen, enabling support for another protocol and URL scheme in QtWebKit is a fairly simple process involving the creation of a network manager and custom reply object. The implementation challenges are mostly related to how the protocol is handled by the custom subclass where, in our case, we need to issue the appropriate commands in the correct order to obtain data from the FTP server.

We also need to ensure that that the reply emits the appropriate signals to inform the browser that it has content to be read. Our implementation is intentionally simple, only notifying the browser with the readyRead() signal when all the content is ready to read and emitting the signal to indicate that communication is complete — a more sophisticated approach would interleave the commands sent to the server with the emission of signals, allowing the browser to read content as data is obtained from the FTP server.

The reply also needs to be open for reading. Forgetting to call the open() function is a common error to make when dealing with devices, but in this case it is the reply's responsibility to open itself. It must indicate how much content it has for the browser to read. As we have seen, this is done by setting the reply's header with the appropriate value. With this information available, the browser can read from the reply when the content becomes available, displaying a directory listing or downloading content depending on the type of data supplied.

We can use the approach described in this article to enable support for other protocols by writing or extending a network manager to handle URL schemes such as mailto, sip, news, file and ldap. Applications that integrate Web content with information from other sources can also provide custom URL schemes as long as care is taken not to use an existing public scheme.

The source code for the example discussed in this article is available for download:

David Boddie is a technical writer working in Nokia Qt Development Frameworks in Oslo, Norway.