ffmpeg - tutorial01

Tutorial 01: Making Screencaps

Overview

Movie files have a few basic components. First, the file itself is called a container, and the type of container determines where the information in the file goes. Examples of containers are AVI and Quicktime. Next, you have a bunch of streams; for example, you usually have an audio stream and a video stream. (A "stream" is just a fancy word for "a succession of data elements made available over time".) The data elements in a stream are called frames. Each stream is encoded by a different kind of codec. The codec defines how the actual data is COded and DECoded - hence the name CODEC. Examples of codecs are DivX and MP3. Packets are then read from the stream. Packets are pieces of data that can contain bits of data that are decoded into raw frames that we can finally manipulate for our application. For our purposes, each packet contains complete frames, or multiple frames in the case of audio.

At its very basic level, dealing with video and audio streams is very easy:

10 OPEN video_stream FROM video.avi
20 READ packet FROM video_stream INTO frame
30 IF frame NOT COMPLETE GOTO 20
40 DO SOMETHING WITH frame
50 GOTO 20

Handling multimedia with ffmpeg is pretty much as simple as this program, although some programs might have a very complex "DO SOMETHING" step. So in this tutorial, we're going to open a file, read from the video stream inside it, and our DO SOMETHING is going to be writing the frame to a PPM file.

Opening the File

First, let's see how we open a file in the first place. With ffmpeg, you have to first initialize the library. (Note that some systems might have to use and instead.)

#include 
#include 
...
int main(int argc, charg *argv[]) { ();

This registers all available file formats and codecs with the library so they will be used automatically when a file with the corresponding format/codec is opened. Note that you only need to call () once, so we do it here in main(). If you like, it's possible to register only certain individual file formats and codecs, but there's usually no reason why you would have to do that.

Now we can actually open the file:

 *pFormatCtx;

// Open video file
if((&pFormatCtx, argv[1], NULL, 0, NULL)!=0)
  return -1; // Couldn't open file

We get our filename from the first argument. This function reads the file header and stores information about the file format in the  structure we have given it. The last three arguments are used to specify the file format, buffer size, and format options, but by setting this to NULL or 0, libavformat will auto-detect these.

This function only looks at the header, so next we need to check out the stream information in the file.:

// Retrieve stream information
if((pFormatCtx)<0)
  return -1; // Couldn't find stream information

This function populates pFormatCtx->streams with the proper information. We introduce a handy debugging function to show us what's inside:

// Dump information about file onto standard error
dump_format(pFormatCtx, 0, argv[1], 0);

Now pFormatCtx->streams is just an array of pointers, of size pFormatCtx->nb_streams, so let's walk through it until we find a video stream.

int i;  *pCodecCtx;

// Find the first video stream
videoStream=-1;
for(i=0; inb_streams; i++)
  if(pFormatCtx->streams[i]->codec->codec_type==CODEC_TYPE_VIDEO) {
    videoStream=i;
    break;
  }
if(videoStream==-1)
  return -1; // Didn't find a video stream

// Get a pointer to the codec context for the video stream
pCodecCtx=pFormatCtx->streams[videoStream]->codec;

The stream's information about the codec is in what we call the "codec context." This contains all the information about the codec that the stream is using, and now we have a pointer to it. But we still have to find the actual codec and open it:

AVCodec *pCodec;

// Find the decoder for the video stream
pCodec=(pCodecCtx->codec_id);
if(pCodec==NULL) {
  fprintf(stderr, "Unsupported codec!\n");
  return -1; // Codec not found
}
// Open codec
if((pCodecCtx, pCodec)<0)
  return -1; // Could not open codec

Some of you might remember from the old tutorial that there were two other parts to this code: addingCODEC_FLAG_TRUNCATED to pCodecCtx->flags and adding a hack to correct grossly incorrect frame rates. These two fixes aren't in ffplay.c anymore, so I have to assume that they are not necessary anymore. There's another difference to point out since we removed that code: pCodecCtx->time_base now holds the frame rate information. time_base is a struct that has the numerator and denominator (). We represent the frame rate as a fraction because many codecs have non-integer frame rates (like NTSC's 29.97fps).

Storing the Data

Now we need a place to actually store the frame:

 *pFrame;

// Allocate video frame
pFrame=();

Since we're planning to output PPM files, which are stored in 24-bit RGB, we're going to have to convert our frame from its native format to RGB. ffmpeg will do these conversions for us. For most projects (including ours) we're going to want to convert our initial frame to a specific format. Let's allocate a frame for the converted frame now.

// Allocate an  structure
pFrameRGB=();
if(pFrameRGB==NULL)
  return -1;

Even though we've allocated the frame, we still need a place to put the raw data when we convert it. We use to get the size we need, and allocate the space manually:

uint8_t *buffer;
int numBytes;
// Determine required buffer size and allocate buffer
numBytes=(PIX_FMT_RGB24, pCodecCtx->width,
                            pCodecCtx->height);
buffer=(uint8_t *)(numBytes*sizeof(uint8_t));

 is ffmpeg's malloc that is just a simple wrapper around malloc that makes sure the memory addresses are aligned and such. It will not protect you from memory leaks, double freeing, or other malloc problems.

Now we use  to associate the frame with our newly allocated buffer. About the  cast: the AVPicture struct is a subset of the  struct - the beginning of the AVFrame struct is identical to the AVPicture struct.

// Assign appropriate parts of buffer to image planes in pFrameRGB
// Note that pFrameRGB is an , but AVFrame is a superset
// of  (( *)pFrameRGB, buffer, PIX_FMT_RGB24,
                pCodecCtx->width, pCodecCtx->height);

Finally! Now we're ready to read from the stream!

Reading the Data

What we're going to do is read through the entire video stream by reading in the packet, decoding it into our frame, and once our frame is complete, we will convert and save it.

int frameFinished;  packet;

i=0;
while((pFormatCtx, &packet)>=0) {
  // Is this a packet from the video stream?
  if(packet.stream_index==videoStream) {
	// Decode video frame (pCodecCtx, pFrame, &frameFinished,
                         packet.data, packet.size);
    
    // Did we get a video frame?
    if(frameFinished) {
    // Convert the image from its native format to RGB (( *)pFrameRGB, PIX_FMT_RGB24, 
            (*)pFrame, pCodecCtx->pix_fmt, 
			pCodecCtx->width, pCodecCtx->height);
	
        // Save the frame to disk
        if(++i<=5)
          SaveFrame(pFrameRGB, pCodecCtx->width, 
                    pCodecCtx->height, i);
    }
  }
    
  // Free the packet that was allocated by  (&packet);
}

The process, again, is simple: () reads in a packet and stores it in the  struct. Note that we've only allocated the packet structure - ffmpeg allocates the internal data for us, which is pointed to by packet.data. This is freed by the () later. () converts the packet to a frame for us. However, we might not have all the information we need for a frame after decoding a packet, so avcodec_decode_video() sets frameFinished for us when we have the next frame. Finally, we use () to convert from the native format (pCodecCtx->pix_fmt) to RGB. Remember that you can cast an  pointer to an  pointer. Finally, we pass the frame and height and width information to our SaveFrame function.

A note on packets 

	Technically a packet can contain partial frames or other bits of data, but ffmpeg's parser ensures that the packets we get contain either complete or multiple frames.

Now all we need to do is make the SaveFrame function to write the RGB information to a file in PPM format. We're going to be kind of sketchy on the PPM format itself; trust us, it works.

void SaveFrame( *pFrame, int width, int height, int iFrame) {

  FILE *pFile;
  char szFilename[32];
  int  y;
  
  // Open file
  sprintf(szFilename, "frame%d.ppm", iFrame);
  pFile=fopen(szFilename, "wb");
  if(pFile==NULL)
    return;
  
  // Write header
  fprintf(pFile, "P6\n%d %d\n255\n", width, height);
  
  // Write pixel data
  for(y=0; ydata[0]+y*pFrame->linesize[0], 1, width*3, pFile);
  
  // Close file
  fclose(pFile);
}

We do a bit of standard file opening, etc., and then write the RGB data. We write the file one line at a time. A PPM file is simply a file that has RGB information laid out in a long string. If you know HTML colors, it would be like laying out the color of each pixel end to end like #ff0000#ff0000.... would be a red screen. (It's stored in binary and without the separator, but you get the idea.) The header indicated how wide and tall the image is, and the max size of the RGB values.

Now, going back to our main() function. Once we're done reading from the video stream, we just have to clean everything up:

// Free the RGB image (buffer); (pFrameRGB);

// Free the YUV frame (pFrame);

// Close the codec
avcodec_close(pCodecCtx);

// Close the video file (pFormatCtx);

return 0;

You'll notice we use  for the memory we allocated with avcode_alloc_frame and .

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
编译
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
//ffmpeg
tar -xf ffmpeg-2.0.1.tar.bz2
./configure --prefix=/opt/libffmpeg/host --disable-yasm --enable-static
make && make install

//tutorial01.c
gcc -o tutorial01 tutorial01.c -lavutil -lavformat -lavcodec -lswscale -lz -lm -L/opt/libffmpeg/host/lib/ -I/opt/libffmpeg/host/include
./tutorial01 /root/test.avi || ./tu rtsp://192.168.1.1/test_stream

// tutorial01.c
//
// This tutorial was written by Stephen Dranger ([e-mail]).
//
// Code based on a tutorial by Martin Bohme ([e-mail])
// Tested on Gentoo, CVS version 5/01/07 compiled with GCC 4.1.1

// A small sample program that shows how to use libavformat and libavcodec to
// read video from a file.
//
// Use the Makefile to build all examples.
//
// Run using
//
// tutorial01 myvideofile.mpg
//
// to write the first five frames from "myvideofile.mpg" to disk in PPM
// format.
//
//
//gcc -o tutorial01 tutorial01.c -lavutil -lavformat -lavcodec -lswscale -lz -lavutil -lm -L/opt/libffmpeg/host/lib/ -I/opt/libffmpeg/host/include
//./tutorial01 /opt/ffmpeg-2.0.1/test.mp4
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

#include <stdio.h>

void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame)
{
    // Open file
    char szFilename[32];
    sprintf(szFilename, "frame%d.ppm", iFrame);
    FILE *pFile = fopen(szFilename, "wb");
    if(pFile==NULL) return;

    // Write header
    fprintf(pFile, "P6\n%d %d\n255\n", width, height);

    // Write pixel data
    int y;
    for(y=0; y<height; y++)
        fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);
    /**
    RGB24，故每个像素点3字节
    fwrite函数每次把横向width个像素即width * 3个字节写入文件，循环height次即可把整张图片数据写入到文件，即形成一张图片。
    */

    // Close file
    fclose(pFile);
}

int main(int argc, char *argv[])
{
    AVFormatContext *pFormatCtx = NULL;
    AVCodecContext *pCodecCtx = NULL;
    AVCodec *pCodec = NULL;
    AVFrame *pFrame = NULL;
    AVFrame *pFrameRGB = NULL;
    AVPacket        packet;
    int frameFinished;
    int i, videoStream;
    int numBytes;
    uint8_t *buffer = NULL;
    AVDictionary *optionsDict = NULL;
    struct SwsContext *sws_ctx = NULL;

    if(argc < 2) {
        printf("Please provide a movie file\n");
        return -1;
    }

    // Register all formats and codecs
    av_register_all();

    // Open video file
    if(avformat_open_input(&pFormatCtx, argv[1], NULL, NULL)!=0)
        return -1; // Couldn't open file

    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0)
        return -1; // Couldn't find stream information

    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);

    // Find the first video stream
    videoStream=-1;
    for(i=0; i<pFormatCtx->nb_streams; i++){
        if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
            videoStream=i;
            break;
        }
    }

    if(videoStream==-1)
        return -1; // Didn't find a video stream

    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;

    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
        fprintf(stderr, "Unsupported codec!\n");
        return -1; // Codec not found
    }

    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec, &optionsDict)<0)
        return -1; // Could not open codec

    // Allocate video frame
    pFrame=av_frame_alloc();
    /**
    存放h264解码后的yuv图像
    pFrame->data[0]指向Y向量
    每pFrame->linesize[0]个字节存放width个字节的数据。 例如：pFrame->linesize[0] = 1952， width = 1920的意思是pFrame->data[0]值向的数据以1952字节为一组，每一组的前1920个字节为Y向量的数据， 共有height组
    pFrame->data[1]指向U向量
    每pFrame->linesize[1]个字节存放width / 2个字节的数据。 例如：pFrame->linesize[1] = 976， width / 2 = 960的意思是pFrame->data[1]值向的数据以976字节为一组，每一组的前960个字节为U向量的数据， 共有height / 4组
    pFrame->data[2]指向V向量
    每pFrame->linesize[2]个字节存放width / 2个字节的数据。 例如：pFrame->linesize[2] = 976， width / 2 = 960的意思是pFrame->data[2]值向的数据以976字节为一组，每一组的前960个字节为V向量的数据， 共有height / 4组

    YUV图片即是把Y，U，V向量的数据按顺序保存为文件。 例如一张width * height的YUV420图片，前width * height字节存放Y向量数据，接下来的width * height / 4字节存放U向量数据， 再下来的width * height / 4字节存放V向量数据
    共 width * height * 1.5个字节数据。相对RGB24图片少了一半的大小，如果是YUV444的图片，大小就跟RGB24一致了。
    */

    // Allocate an AVFrame structure
    pFrameRGB=av_frame_alloc();
    /**
    存放yuv图像转换后的rgb24图像
    如果图像宽高为 width height， 则rgb24图像大小为 width * height * 3
    */
    if(pFrameRGB==NULL)
        return -1;

    // Determine required buffer size and allocate buffer
    numBytes=avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height);
    buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));

    sws_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height, PIX_FMT_RGB24, SWS_BILINEAR, NULL, NULL, NULL);

    // Assign appropriate parts of buffer to image planes in pFrameRGB
    // Note that pFrameRGB is an AVFrame, but AVFrame is a superset
    // of AVPicture
    avpicture_fill((AVPicture *)pFrameRGB, buffer, PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height);

    // Read frames and save first five frames to disk
    i=0;
    while(av_read_frame(pFormatCtx, &packet)>=0) {
        // Is this a packet from the video stream?
        if(packet.stream_index==videoStream) {
            // Decode video frame
            avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);

            // Did we get a video frame?
            if(frameFinished) {
                // Convert the image from its native format to RGB
                sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);

                // Save the frame to disk
                if(++i<=5)
                    SaveFrame(pFrameRGB, pCodecCtx->width, pCodecCtx->height, i);
            }
        }
        // Free the packet that was allocated by av_read_frame
        av_free_packet(&packet);
    }

    // Free the RGB image
    av_free(buffer);
    av_free(pFrameRGB);

    // Free the YUV frame
    av_free(pFrame);

    // Close the codec
    avcodec_close(pCodecCtx);

    // Close the video file
    avformat_close_input(&pFormatCtx);

    return 0;
}