HTTP Client Library (libcurl)

Table of Contents

Introduction

The Cafe SDK provides HTTP client APIs as a subset of the popular multi-protocol library libcurl (version 7.21.7). Both the easy and multi interfaces of libcurl are supported.

NOTE:
The behavior of the Cafe curl API is identical to other platforms except for situations that are described in Exception section and Limitations section. Modified version of libcurl (version 7.21.7) documentation is made available that includes the Cafe subset of full API and options supported.

Getting Started

To use the HTTP client, first make sure that network configuration is set and desired network interface is UP.

After the network comes up, initialize the socket library using SOInit before using any of the HTTP client functions. Also use SOFinish to cleanup the socket library after the application has finished.

For information on the AC and Socket Library functions, see the Network section.

Header and Library Files

Include the following header files.

HTTP Client Programming

For information on programming using libcurl APIs, see libcurl tutorial. The behavior of the curl APIs is identical to other platforms except for the following limitations and situations which are described in Exception section.

Limitations

The current version of the HTTP client library available in the Cafe SDK has the following limitations.

  1. Only the HTTP (and HTTPS) protocol is currently enabled
  2. Compressed transfer encodings are not yet supported
  3. Provide mutex callbacks if you are using the "share" interface in a multithreaded application. See the libcurl-share and curl_share_setopt sections of the libcurl documentation for details.
  4. Only synchronous DNS lookup is supported. Timeouts and progress callbacks are not called during DNS lookup stage. Also multi handles block during DNS lookup stage.
  5. curl_getenv always returns NULL
  6. All the API functions except the following are supported: curl_multi_socket, curl_multi_socket_action, curl_multi_assign.
  7. Features that access the file system will not work. Some examples of such features are:
    • Persistent cookies (CURLOPT_COOKIEFILE, CURLOPT_COOKIEJAR) are not supported.
    • Adding form data using file (using option CURLFORM_FILECONTENT) is not supported.
    • Sending data using CURLOPT_READDATA to point to FILE* is not supported.
    • Saving data using CURLOPT_WRITEDATA to point to FILE* is not supported.
  8. The CURLINFO_CERTINFO option is not supported in the curl_easy_getinfo call.

Memory Management

By default, the HTTP client library uses standard memory management functions like malloc and free. As explained in The Default Heap section, these functions use whatever heap is set as the default. Applications can choose to specify their own custom memory management functions to be used by the HTTP client library by using the curl_global_init_mem API (as demonstrated in the HTTP client library easy interface demo program below). The HTTP client library uses name resolver from Cafe Socket APIs, use SOResolverRegisterAllocator to set the memory management functions for the resolver.

Performance

The Cafe libcurl implementation introduces an option for changing the internal socket receive buffer size and enabling the window scaling in libcurl. It is provided as an option of curl_easy_setopt function.

CURLOPT_SO_WINSCALE Enables/Disables window scaling of the internal socket in libcurl. Set 1 to the parameter to enable window scaling, otherwise set 0 to disable it.
CURLOPT_SO_RCVBUF Configures socket receive buffer size of the internal socket in libcurl. Set socket receive buffer size to the parameter. The maximum number of the parameter is 65535. If the window scaling is enabled, then the maximum number of the parameter is 128K(1024*128).

Again, to set 128K receive buffer, options must be set in the following order.

{
u32 winscale = 1;
u32 recv_buff_size = 1024*128;

/* Set Window Scaling */
curl_easy_setopt(easy_handle, CURLOPT_SO_WINSCALE, winscale);

/* Set Receive Buffer Size */
curl_easy_setopt(easy_handle, CURLOPT_SO_RCVBUF, recv_buff_size);
}

CURLOPT_SOCKOPTFUNCTION is also available so the same option can be set by using this option and socket options. Refer to libcurl documentation for further information.

Timeout

The behavior of libcurl's timeout is different from the Nintendo Wii HTTP library. By default, curl_easy_perform will not abort even if the data transfer is stalled for any reasons (e.g. public network going down or server not sending data). Applications are required to handle detection of stalled transfers (and abort the requests).

libcurl provides a way to handle a timeout in such cases as a subset of curl_easy_setopt. There are some options which can be configured with curl_easy_setopt. Refer to curl_easy_setopt for further information.

HTTPS

Use the SSL library to setup a SSL context to access HTTPS URLs. Initialize the SSL library using NSSLInit after SOInit, cleanup the SSL library by using NSSLFinish after the application has finished. Create NSSLContext using NSSLCreateContext by specifying the desired version of SSL (SSL 3.0 and TLS 1.0 are supported), destroy the context using NSSLDestroyContext after the application has finished. Use NSSLAddServerPKIExternal to specify the trusted CA certificates for validating the peer certificates (optional). Use NSSLAddCRLExternal to specify the Certificate Revocation List (CRL) (optional). This function can be called multiple times to add multiple CRLs. If CRLs have been specified, use NSSLContextSetFlags to turn on CRL checking by setting the appropriate flags. Use NSSLSetClientPKIExternal to specify the client certificate and private key if the server performs client validation (optional). Attach the context to curl easy handle using curl_easy_setopt using OPTION CURLOPT_NSSL_CONTEXT. Also use OPTION CURLOPT_NSSL_PEER_VERIFY_OPT to specify how you would like to verify the peer certificate (default value NSSL_VERIFY_ALL_EXCEPT_DATE is used if CURLOPT_NSSL_PEER_VERIFY_OPT is not set on a curl easy handle). curl_easy_getinfo with CURLINFO_SSL_VERIFYRESULT returns detailed error code for peer certificate validation (NSSL_CERT_VALID or NSSL_CERT_V_ERR* from cafe/nssl/nsslclient.h. See Cafe SSL library for details.

Exception

Proxy Settings

Cafe SDK's libcurl is modified to automatically read and apply proxy settings. Proxy settings are configured by System Config Tool and stored to the NAND. If there is an available proxy settings in the NAND, those settings are automatically applied for every libcurl data transfer. Followings are parameters which are automatically set when curl_easy_init function gets called.

CURLOPT_PROXY HTTP proxy address.
CURLOPT_NOPROXY Hostnames which do not require a proxy to get reached.
CURLOPT_PROXYUSERNAME User name for the proxy authentication.
CURLOPT_PROXYPASSWORD Password for the proxy authentication.
CURLOPT_PROXYPORT Proxy port number to connect.
CURLOPT_PROXYAUTH Proxy authentication method.

An applications which want to overwrite the proxy settings are required to provide all proxy settings which are listed above with curl_easy_setopt function. See documentation of curl_easy_setopt function in libcurl C API documentation for details of these proxy options.

NOTE:
Proxy settings are loaded from the NAND only at the first call of curl_global_init function and cached in the libcurl. This means even if the system proxy settings (which are in the system memory, not in the NAND) are changed by calling AC APIs from an application after curl_global_init function gets called, it is not applied to the curl data transfer automatically (curl_easy_init function obtains the proxy settings from the cache which are inside the libcurl and applies it to every handle). To apply the latest proxy settings, an application have to call curl_global_cleanup function and call curl_global_init function again then it gets applied to a subsequent curl data transfer.

curl_global_init

curl_global_init no longer returns Cafe-specific error codes even if it fails to read proxy config from the NAND. The stored proxy in the NAND is not used for HTTP data transfer in such a situation.

Demo Programs

NameHTTP client library easy interface demo
Files curl/easy/easy_demo.c
Description This example demonstrates how to fetch an HTTP GET URL using the HTTP client library. The example first waits till the network comes up using nn::ac::IsConnected. At this point, Cafe is provisioned to start using the socket library for data exchange. It first initializes the socket library using SOInit.

For HTTPS transfers, it initializes the SSL library using NSSLInit, creates a SSL context using NSSLCreateContext. It then specifies client and server certificates to be used (if any) using NSSLSetClientPKIExternal and NSSLAddServerPKIExternal. It also adds Certificate Revocation Lists (CRL) (if any) using NSSLAddCRLExternal and turns on CRL checking using NSSLContextSetFlags.

A URL to be fetched is extracted from the first argument to the program. The URL is fetched using the following steps.
  • Initializes HTTP client library using curl_global_init (using curl_global_init_mem instead, if custom memory management functions are to be used).
  • Creates an easy handle to perform the HTTP transfer using curl_easy_init.
  • Sets the URL and other options on the easy handle using a series of calls to curl_easy_setopt. Sets up the callback function curl_write_function to handle the data received from the server.
  • Performs the HTTP transfer using curl_easy_perform; this will block till the transfer is completed. Callback function curl_write_function will be called for each chunk of data received from the server. The function writes the same to the output port using OSReport.
  • Cleans up the easy handle using curl_easy_cleanup.
  • Cleans up the HTTP client library using curl_global_cleanup.
  • Destroys the SSL context using NSSLDestroyContext.
  • Cleans up the SSL library using NSSLFinish.
  • Cleans up the socket library using SOFinish.
  • Cleans up the AC library using nn::ac::Finalize.
Setup
  • CAT-DEV
  • Wireless AP (DHCP server enabled) or USB-Ethernet dongle
  • HTTP server reachable from the Wireless AP or ethernet.
For HTTPS with client verification, specify appropriate certificate (CLIENT_CERT_DER_BUFF, in DER format), private key (PKCS1_RSA_PVT_KEY_BUFF, in PKCS1 format) and define CURL_EASY_DEMO_USE_CLIENT_CERT. For peer certificate verification, specify appropriate certificate (SERVER_CERT_DER_BUFF, in DER format) and define CURL_EASY_DEMO_USE_SERVER_CERT.

Run easy_demo program on Cafe and provide the URL to be fetched as a command line argument (for example, caferun -a "http://myhost.com/index.html" easy_demo.rpx). The response obtained from the server will be written to the output port.
NOTE:
The specified URL should be reachable from the wireless AP or ethernet.

NameHTTP client library multi interface demo
Filescurl/multi/multi_demo.c
Description This example demonstrates how to fetch multiple HTTP GET URLs in parallel using HTTP client library. The example first waits till the network comes up using nn::ac::IsConnected. At this point, Cafe is provisioned to start using the socket library for data exchange. It first initializes the socket library using SOInit.

For HTTPS transfers, it initializes the SSL library using NSSLInit, creates a SSL context using NSSLCreateContext. It then specifies client and server certificates to be used (if any) using NSSLSetClientPKIExternal and NSSLAddServerPKIExternal.

The URLs to be fetched are extracted from the arguments to the program. The URLs are fetched using the following steps.
  • Initializes the HTTP client library using curl_global_init.
  • Creates easy handles using curl_easy_init once for each of the URLs to be fetched.
  • Sets the URL and other options on the easy handle using a series of calls to curl_easy_setopt. Sets up the callback function curl_write_function to handle the data received from the server. Adds the easy handle to the curl_easy_handles array.
  • Creates a multi handle using curl_multi_init.
  • Adds the easy handles from the curl_easy_handles array to the multi handle using curl_add_multi_handle.
  • Calls curl_multi_perform repeatedly till all the transfers are completed. The callback function curl_write_function will be called for each chunk if data received from the server. The function writes the same to the output port using OSReport.
  • Removes the easy handles from the multi handle using curl_multi_remove_handle.
  • Cleans up the easy handles using curl_easy_cleanup.
  • Cleans up the multi handle using curl_multi_cleanup.
  • Cleans up the HTTP client library using curl_global_cleanup.
  • Destroys the SSL context using NSSLDestroyContext.
  • Cleans up the SSL library using NSSLFinish.
  • Cleans up the socket library using SOFinish.
  • Cleans up the AC library using nn::ac::Finalize.
Setup
  • CAT-DEV
  • Wireless AP (DHCP server enabled) or USB-Ethernet dongle
  • HTTP server reachable from the Wireless AP or ethernet.
For HTTPS with client verification, specify appropriate certificate (CLIENT_CERT_DER_BUFF, in DER format), private key (PKCS1_RSA_PVT_KEY_BUFF, in PKCS1 format) and define CURL_EASY_DEMO_USE_CLIENT_CERT. For peer certificate verification, specify appropriate certificate (SERVER_CERT_DER_BUFF, in DER format) and define CURL_EASY_DEMO_USE_SERVER_CERT.

Run the multi_demo program on Cafe and provide the URLs to be fetched as a command line argument (for example, caferun -a "http://myhost.com/index.html,http://myhost2.com/index.html" multi_demo.rpx). The response obtained from the servers will be written to the output port.
NOTE:
The specified URLs should be reachable from the wireless AP or ethernet.

NameSpeed Test
Filesspeedtest/speedtest.cpp
Description This example demonstrates download network performance with multiple simultaneous connections and custom socket options using LibCurl's multi-interface. The example first waits till the network comes up using nn::ac::IsConnected. At this point, Cafe is provisioned to start using the socket library for data exchange. It first initializes the socket library using SOInit
Setup
  • CAT-DEV v3 or later.
  • Wireless AP (DHCP server enabled) or USB-Ethernet dongle
  • HTTP server reachable from the Wireless AP or ethernet.
Run the speedtest program on Cafe and provide the URL and other parameters as a command line argument (for example, caferun -a "http://myhost.com/index.html 1 1" speedtest.rpx). The response obtained from the server will be written to the output port.

Usage
caferun -a "<url> <num-sim-con> <num-downloads>" speedtest.rpx
  • <url> - The HTTP URL address of the resource to download. Defaults to "http://ipv4.download.thinkbroadband.com/100MB.zip"
  • <num-sim-con> - How many simultaneous connections to use when downloading. Defaults to "1"
  • <num-downloads> - How many times to download the resource. Defaults to "1".
    NOTE:
    The program will average the results for a more stable result.

Do Not Call From

Do not call any of the HTTP client APIs from callbacks.

See Also

Network Overview
libcurl API Functions

Revision History

2014/06/02 Added NSSL CRL API usage details in the curl easy_demo program.
2014/05/14 Added speedtest demo description.
2013/09/12 Converted.
2013/08/29 Added information in Limitations section to indicate CURLINFO_CERTINFO is not supported in curl_easy_getinfo.
2013/07/30 Fixed dead html links.
2013/05/06 curl_global_init doesn't return an error anymore even when it fails to read proxy.
2013/03/18 Added description about the return code of curl_global_init.
2013/01/23 Updates for System Config Tool.
2012/10/01 Added description about timeout.
2012/09/18 Added description about performance.
2012/07/09 Referred about automatic reading of proxy settings.
2012/06/08 Replaced the description related to NC APIs with AC APIs.
2011/09/29 Initial version.


CONFIDENTIAL