This section describes the contents of the H.264/MPEG-4 AVC Decoder library and its application programming interface.
The H.264 Decoder library provides a programming interface for decoding H.264 video data. The library can accept a bitstream that complies with the H.264/MPEG-4 AVC video format and output decoded image data.
The package content of the HE-AAC library is as follows.
|Path and Filename||Description|
||The H.264/MPEG-4 AVC Decoder library itself.|
||The header file that defines parameters, error values, function names, and other values.|
This library uses the following resources.
|Description||Required Size (KB)|
|User application memory used by this library.||See the section on Memory.|
User applications need to allocate memory for the library to use. For information and restrictions on memory allocation, see the section on Memory.
This library provides features for decoding data from H.264/MPEG-4 AVC Video Elementary Streams and has the following specifications.
Input streams have the following formats and image sizes. Non-standard streams may not be supported.
Input data has the following format.
Supported profiles and levels:
Profiles: Constrained Baseline Profile, Main Profile, and High Profile.
Levels: Level 1 through Level 4.1
Input images have the following sizes.
However, real-time performance may not be expected when using a stream or camera images that exceed 1080p, 24fps.
The maximum stream size per frame is limited as follows.
The following output format is supported.
User applications need to allocate the memory that the library will use.
The library uses three types of memory: work memory, frame memory, and memory for the input bitstream.
|Type of Memory||Required Size (bytes)|
|Frame memory||3,342,336 * n|
The size of work memory and frame memory required depend on the profile identifier, level, image
size, and other usage conditions. Use a utility function
H264DECMemoryRequirement to get the
required memory size.
You need enough frame memory for at least six frames: one frame is used for decoding and five frames are used for the library's internal buffers. A ring buffer is normally created when the decoder outputs synchronized audio and video. Increase the amount of frame memory as needed by the user application.
If you use output from the MP4 demultiplexer or a buffer that was loaded from a file, you do not need to provide any bitstream memory in advance.
The following restrictions apply to the memory used by the library.
H264DECInitParamfunction) until it is shut down (using the
H264DECClosefunction). Bitstream memory must be valid from the point that stream input starts (using the
H264DECSetBitstreamfunction) until decoding ends (using the
This library decodes data one frame at a time.
Because the MP4 demultiplexer outputs one frame at a time, your application can send that output directly to this library and decode it.
To decode a Video Elementary Stream, use the utility function
this library to get the data size per-frame.
The configured callback function gets decoding results from the library.
The library output (the
H264DECOutput data structure) passed to the callback function, including the
output decoded data and VUI (voice-user interface) output within it, is valid until the callback function
completes. To reference this data outside of the callback function, it must be duplicated explicitly within
the callback function. However, the location indicated by Result within the decoded data output (the
actual decoded output image data) is maintained after the callback function completes, so only the
pointer in Result needs to be duplicated. This library does not overwrite the location indicated by Result
after invoking the callback function.
To access the memory used in the callback function with this library, it is possible to set user
memory as a parameter. Set an address in user memory when setting the library parameters
H264DECSetParam from the user application. Parameter settings are in Parameter
Settings and the
H264DECOutput structure is described Library Output.
Callback functions are invoked when the user application invokes the library (
function) or when a stream is terminated
H264DECEnd function. However, the library execution
H264DECExecute function invokes the callback function when the decoding results are
stored in the frame memory. For example, when decoding results cannot be acquired normally and
stored in the frame memory due to an error stream or other reasons, the callback function is not
The library invokes a callback function whenever output images are generated to output the decoded
frame(s). However, exactly when the callback function is invoked is determined by the mode, which is
either sequential output mode or buffering mode (as configured by the library's parameter settings with
In sequential output mode, the callback function is invoked and a decoded frame is output as soon as
a single image (frame) has finished decoding. Use this mode when frames are not reordered
during decoding, such as with a normal Baseline Profile stream, to decode without delays and use only
a small memory footprint. If a stream's frames have been reordered, the frames are
output as reordered, so the user application must restore them to their original chronological order in a
separate step. Because the frames do not accumulate in the library with this mode, the
instruction process to terminate the stream (
H264DECEnd function) does not invoke the callback
In buffering mode, because the frame is output after rearranging into the original video frames order, a
prescribed number of frames (five frames for this library) accumulate in the library. The callback
function is invoked with this mode while invoking the library (
H264DECExecute function) when the
decoding results stored in frame memory are accumulated in the library, and that number exceeds the
prescribed number of frames. That is, when decoding results for a single frame are obtained
normally, the decoding results are accumulated, and when the accumulated decoding results exceed
the prescribed number of frames, the callback function is invoked, and the single frame is output.
Using this mode with a stream whose frames have been reordered, a delay is introduced, but the
decoding results are retrieved in the correct order.
With this mode, when exiting the library (when the
H264DECEnd function is called) to output any
buffered decoding result data, the library may call the callback function a single time to output the
decoding results for the number of the accumulated frames.
This section describes the decoding features of the H.264 decoder library.
The following is a data flow diagram for decoding with the H.264 decoder library.
This library uses a hardware decoder. This reduces the amount of work that the CPU needs to do, but also means that the library waits for the hardware to finish processing. After a library function is called by an application, it will not return for approximately as long as it takes the hardware decoder to process data.
The following table provides more information on the data that is passed between modules.
|1||Decoded H.264/MPEG-4 AVC Video Elementary Stream|
|2||H.264/MPEG-4 AVC frame data|
|3||Raw (NV12) video data|
The following figure shows the basic control flow for decoding with the H.264 decoder library. As shown in the diagram, callback functions are invoked to get decoded results when the library is
H264DECExecute function call) or a stream is terminated (the
H264DECEnd function call).
If you are decoding video data for normal playback, you repeat the bitstream set operation
H264DECSetBitstream function and the execute instruction
H264DECExecute on a continuous
stream. If you want to start decoding from a different position in the same stream (including recovery and playback from an error stream), or if you want to decode a different stream, you need to take the
H264DECExecute) completes, terminate the stream (call the
H264DECEndfunction) and then restart it (call the Start Decoding
H264DECExecute) completes, terminate it (call the
H264DECEndfunction), finalize the library (call the
H264DECClosefunction), and finally reopen the library (call the
H264DECOpenfunction) to resume.
By following the listed nine steps, user applications can use the H.264 decoder library to decode streams.
H264DECInitParam- Set the work memory and initialize the library.
H264DECSetParam- Set callback function pointers and other parameters.
H264DECOpen- Initialize the library's internal state according to parameter settings. (To decode a different stream, resume by calling the
H264DECOpenfunction after terminating the current stream (call
H264DECEnd) and closing the library (call
H264DECBegin- Send an instruction to start decoding. (To resume decoding from a different position in the same stream, start the library (call the
H264DECBeginfunction) after terminating the current stream (call the
H264DECSetBitstream- Set the bitstream to decode.
H264DECExecute- Decode the bitstream that you have set.
H264DECEnd- Notify the library to end decoding (free up hardware resources, flush and perform other processes on frames maintained internally).
H264DECFlush- Notify the library to flush the frame it maintains internally.
H264DECClose- Finalize the library.
The library provides the following utility functions to make memory usage more efficient and to decode without an MP4 demultiplexer.
H264DECMemoryRequirement- Calculate the size of work memory and frame memory based on the profile identifier, level, and image size.
H264DECGetImageSize- Get the size of images from a Video Elementary Stream to decode.
H264DECFindDecstartpoint- Find a location from which to start decoding in the input data (the starting position of the sequence parameter set, or SPS1 - contains information related to the overall encoding of the profile, level, and stream). To decode a bitstream, first call this utility function and then start decoding from the position it returns.
H264DECFindIdrpoint- Find a location to resume decoding (the starting position of an IDR frame2 - a frame that allows random access. This frame allows you to properly decode all frames after it even if you never used information for any frames before it) in the input data. This allows you to get a position at which to start playback in the middle of a bitstream (random access).
H264DECCheckDecunitLength- Get the amount of data used as the base unit for decoding.
H264DECCheckMemSegmentation- Determines whether the given memory area is suitable for application as working memory or frame memory.
H264DECCheckSkipableFrame- Determines whether the subsequent decoding is performed correctly even when the target frame is not decoded.
The following figure shows how data is arranged when the output image format is NV12.
ITU-T Recommendation H.264 (2012/01) Advanced video coding for generic audiovisual services
H.264 Sample Demos
2014/07/16 Initial version.