ffmpeg - tutorial06

Tutorial 06: Synching Audio

Synching Audio

So now we have a decent enough player to watch a movie, so let's see what kind of loose ends we have lying around. Last time, we glossed over synchronization a little bit, namely sychronizing audio to a video clock rather than the other way around. We're going to do this the same way as with the video: make an internal video clock to keep track of how far along the video thread is and sync the audio to that. Later we'll look at how to generalize things to sync both audio and video to an external clock, too.

Implementing the video clock

Now we want to implement a video clock similar to the audio clock we had last time: an internal value that gives the current time offset of the video currently being played. At first, you would think that this would be as simple as updating the timer with the current PTS of the last frame to be shown. However, don't forget that the time between video frames can be pretty long when we get down to the millisecond level. The solution is to keep track of another value, the time at which we set the video clock to the PTS of the last frame. That way the current value of the video clock will bePTS_of_last_frame + (current_time - time_elapsed_since_PTS_value_was_set). This solution is very similar to what we did with get_audio_clock.

So, in our big struct, we're going to put a double video_current_pts and a int64_t video_current_pts_time. The clock updating is going to take place in the video_refresh_timer function:

void video_refresh_timer(void *userdata) {

  /* ... */

  if(is->video_st) {
    if(is->pictq_size == 0) {
      schedule_refresh(is, 1);
    } else {
      vp = &is->pictq[is->pictq_rindex];

      is->video_current_pts = vp->pts;
      is->video_current_pts_time = ();

Don't forget to initialize it in stream_component_open:

    is->video_current_pts_time = ();

And now all we need is a way to get the information:

double get_video_clock(VideoState *is) {
  double delta;

  delta = (() - is->video_current_pts_time) / 1000000.0;
  return is->video_current_pts + delta;
}

Abstracting the clock

But why force ourselves to use the video clock? We'd have to go and alter our video sync code so that the audio and video aren't trying to sync to each other. Imagine the mess if we tried to make it a command line option like it is in ffplay. So let's abstract things: we're going to make a new wrapper function, get_master_clock that checks an av_sync_type variable and then call get_audio_clock, get_video_clock, or whatever other clock we want to use. We could even use the computer clock, which we'll call get_external_clock:

enum {
  AV_SYNC_AUDIO_MASTER,
  AV_SYNC_VIDEO_MASTER,
  AV_SYNC_EXTERNAL_MASTER,
};

#define DEFAULT_AV_SYNC_TYPE AV_SYNC_VIDEO_MASTER

double get_master_clock(VideoState *is) {
  if(is->av_sync_type == AV_SYNC_VIDEO_MASTER) {
    return get_video_clock(is);
  } else if(is->av_sync_type == AV_SYNC_AUDIO_MASTER) {
    return get_audio_clock(is);
  } else {
    return get_external_clock(is);
  }
}
main() {
...
  is->av_sync_type = DEFAULT_AV_SYNC_TYPE;
...
}

Synchronizing the Audio

Now the hard part: synching the audio to the video clock. Our strategy is going to be to measure where the audio is, compare it to the video clock, and then figure out how many samples we need to adjust by, that is, do we need to speed up by dropping samples or do we need to slow down by adding them?

We're going to run a synchronize_audio function each time we process each set of audio samples we get to shrink or expand them properly. However, we don't want to sync every single time it's off because process audio a lot more often than video packets. So we're going to set a minimum number of consecutive calls to the synchronize_audio function that have to be out of sync before we bother doing anything. Of course, just like last time, "out of sync" means that the audio clock and the video clock differ by more than our sync threshold.

So we're going to use a fractional coefficient, say c, and So now let's say we've gotten N audio sample sets that have been out of sync. The amount we are out of sync can also vary a good deal, so we're going to take an average of how far each of those have been out of sync. So for example, the first call might have shown we were out of sync by 40ms, the next by 50ms, and so on. But we're not going to take a simple average because the most recent values are more important than the previous ones. So we're going to use a fractional coefficient, say c, and sum the differences like this:diff_sum = new_diff + diff_sum*c. When we are ready to find the average difference, we simply calculate avg_diff = diff_sum * (1-c).

Note: What the heck is going on here? This equation looks like magic! Well, it's basically a weighted mean using a geometric series as weights. I don't know if there's a name for this (I even checked Wikipedia!) but for more info,  (or at

Here's what our function looks like so far:

/* Add or subtract samples to get a better sync, return new
   audio buffer size */
int synchronize_audio(VideoState *is, short *samples,
		      int samples_size, double pts) {
  int n;
  double ref_clock;
  
  n = 2 * is->audio_st->codec->channels;
  
  if(is->av_sync_type != AV_SYNC_AUDIO_MASTER) {
    double diff, avg_diff;
    int wanted_size, min_size, max_size, nb_samples;
    
    ref_clock = get_master_clock(is);
    diff = get_audio_clock(is) - ref_clock;

    if(diff < AV_NOSYNC_THRESHOLD) {
      // accumulate the diffs
      is->audio_diff_cum = diff + is->audio_diff_avg_coef
	* is->audio_diff_cum;
      if(is->audio_diff_avg_count < AUDIO_DIFF_AVG_NB) {
	is->audio_diff_avg_count++;
      } else {
	avg_diff = is->audio_diff_cum * (1.0 - is->audio_diff_avg_coef);

       /* Shrinking/expanding buffer code.... */

      }
    } else {
      /* difference is TOO big; reset diff stuff */
      is->audio_diff_avg_count = 0;
      is->audio_diff_cum = 0;
    }
  }
  return samples_size;
}

So we're doing pretty well; we know approximately how off the audio is from the video or whatever we're using for a clock. So let's now calculate how many samples we need to add or lop off by putting this code where the "Shrinking/expanding buffer code" section is:

if(fabs(avg_diff) >= is->audio_diff_threshold) {
  wanted_size = samples_size + 
  ((int)(diff * is->audio_st->codec->sample_rate) * n);
  min_size = samples_size * ((100 - SAMPLE_CORRECTION_PERCENT_MAX)
                             / 100);
  max_size = samples_size * ((100 + SAMPLE_CORRECTION_PERCENT_MAX) 
                             / 100);
  if(wanted_size < min_size) {
    wanted_size = min_size;
  } else if (wanted_size > max_size) {
    wanted_size = max_size;
  }

Remember that audio_length * (sample_rate * # of channels * 2) is the number of samples in audio_length seconds of audio. Therefore, number of samples we want is going to be the number of samples we already have plus or minus the number of samples that correspond to the amount of time the audio has drifted. We'll also set a limit on how big or small our correction can be because if we change our buffer too much, it'll be too jarring to the user.

Correcting the number of samples

Now we have to actually correct the audio. You may have noticed that our synchronize_audio function returns a sample size, which will then tell us how many bytes to send to the stream. So we just have to adjust the sample size to the wanted_size. This works for making the sample size smaller. But if we want to make it bigger, we can't just make the sample size larger because there's no more data in the buffer! So we have to add it. But what should we add? It would be foolish to try and extrapolate audio, so let's just use the audio we already have by padding out the buffer with the value of the last sample.

if(wanted_size < samples_size) {
  /* remove samples */
  samples_size = wanted_size;
} else if(wanted_size > samples_size) {
  uint8_t *samples_end, *q;
  int nb;

  /* add samples by copying final samples */
  nb = (samples_size - wanted_size);
  samples_end = (uint8_t *)samples + samples_size - n;
  q = samples_end + n;
  while(nb > 0) {
    memcpy(q, samples_end, n);
    q += n;
    nb -= n;
  }
  samples_size = wanted_size;
}

Now we return the sample size, and we're done with that function. All we need to do now is use it:

void audio_callback(void *userdata, Uint8 *stream, int len) {

  VideoState *is = (VideoState *)userdata;
  int len1, audio_size;
  double pts;

  while(len > 0) {
    if(is->audio_buf_index >= is->audio_buf_size) {
      /* We have already sent all our data; get more */
      audio_size = audio_decode_frame(is, is->audio_buf, sizeof(is->audio_buf), &pts);
      if(audio_size < 0) {
	/* If error, output silence */
	is->audio_buf_size = 1024;
	memset(is->audio_buf, 0, is->audio_buf_size);
      } else {
	audio_size = synchronize_audio(is, (int16_t *)is->audio_buf,
				       audio_size, pts);
	is->audio_buf_size = audio_size;

All we did is inserted the call to synchronize_audio. (Also, make sure to check the source code where we initalize the above variables I didn't bother to define.)

One last thing before we finish: we need to add an if clause to make sure we don't sync the video if it is the master clock:

if(is->av_sync_type != AV_SYNC_VIDEO_MASTER) {
  ref_clock = get_master_clock(is);
  diff = vp->pts - ref_clock;

  /* Skip or repeat the frame. Take delay into account
     FFPlay still doesn't "know if this is the best guess." */
  sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay :
                    AV_SYNC_THRESHOLD;
  if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
    if(diff <= -sync_threshold) {
      delay = 0;
    } else if(diff >= sync_threshold) {
      delay = 2 * delay;
    }
  }
}

And that does it! Make sure you check through the source file to initialize any variables that I didn't bother defining or initializing. Then compile it:

gcc -o tutorial06 tutorial06.c -lavutil -lavformat -lavcodec -lz -lm`sdl-config --cflags --libs`

and you'll be good to go.

Next time we'll make it so you can rewind and fast forward your movie.

源代码如下（修改部分代码，加入一些注释）

// tutorial06.c
// A pedagogical video player that really
//
// This tutorial was written by Stephen Dranger ([e-mail]).
//
// Code based on FFplay, Copyright (c) 2003 Fabrice Bellard,
// and a tutorial by Martin Bohme ([e-mail])
// Tested on Gentoo, CVS version 5/01/07 compiled with GCC 4.1.1
//
// Use the Makefile to build all the samples.
//
// Run using
// tutorial06 myvideofile.mpg
//
// to play the video.
//
//
//
//gcc -o tutorial06 tutorial06.c -lavformat -lavcodec -lavutil -lswscale -lz -lm -L/opt/libffmpeg/lib/ -I/opt/libffmpeg/include/ \
-ldl -lpthread -lSDL2 -I/opt/libsdl/include/ -L/opt/libsdl/lib/ -g
//./tutorial06 testvideo.mov


#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavformat/avio.h>
#include <libswscale/swscale.h>
#include <libavutil/avstring.h>
#include <libavutil/time.h>

#include <SDL2/SDL.h>
#include <SDL2/SDL_thread.h>

#ifdef __MINGW32__
#undef main /* Prevents SDL from overriding main() */
#endif

#include <stdio.h>
#include <math.h>

#define SDL_AUDIO_BUFFER_SIZE 1024
#define MAX_AUDIO_FRAME_SIZE 192000

#define MAX_AUDIOQ_SIZE (5 * 16 * 1024)
#define MAX_VIDEOQ_SIZE (5 * 256 * 1024)

#define AV_SYNC_THRESHOLD 0.01
#define AV_NOSYNC_THRESHOLD 10.0

#define SAMPLE_CORRECTION_PERCENT_MAX 10
#define AUDIO_DIFF_AVG_NB 20

#define FF_ALLOC_EVENT (SDL_USEREVENT)
#define FF_REFRESH_EVENT (SDL_USEREVENT + 1)
#define FF_QUIT_EVENT (SDL_USEREVENT + 2)

#define VIDEO_PICTURE_QUEUE_SIZE 1
#define DEFAULT_AV_SYNC_TYPE AV_SYNC_VIDEO_MASTER

SDL_Window    *screen;
SDL_Renderer    *renderer;
#define PRE_WIDTH 904
#define PRE_HEIGHT 600

static int global_readframe_cntout = 0;

typedef struct PacketQueue {
    AVPacketList *first_pkt, *last_pkt;
    int nb_packets;
    int size;
    SDL_mutex *mutex;
    SDL_cond *cond;
} PacketQueue;


typedef struct VideoPicture {
    SDL_Texture *bmp;
    AVFrame *pFrameYUV;
    char *bufpoint;
    double pts;
    int width, height; /* source height & width */
    int allocated;
} VideoPicture;

typedef struct VideoState {
    AVFormatContext *pFormatCtx;
    int videoStream, audioStream;
    int av_sync_type;
    double          external_clock; /* external clock base */
    int64_t         external_clock_time;
    double          audio_clock;
    AVStream *audio_st;
    PacketQueue     audioq;                        /** 音频AVPacket队列 */
    uint8_t         audio_buf[(MAX_AUDIO_FRAME_SIZE * 3) / 2];    /** 解码后的音频数据 */
    unsigned int audio_buf_size;                    /** 解码后音频数据总size */
    unsigned int audio_buf_index;                /** 当前已使用偏移 */
    AVFrame         audio_frame;                    /** 存放解码后的一帧音频 */
    AVPacket        audio_pkt;                    /** 存放当前需要解码的一个音频包 */
    uint8_t *audio_pkt_data;                /** 初始化为指向一个AVPacket的音频帧数据 */
    int audio_pkt_size;                    /** 初始化为一个AVPacket音频帧的数据长度 */
    int audio_hw_buf_size;                /** 用来干啥的??? */
    double          audio_diff_cum;                    /** used for AV difference average computation */
    double          audio_diff_avg_coef;                /** 固定值c,用于加权平均 */
    double          audio_diff_threshold;
    int audio_diff_avg_count;                /** 每AUDIO_DIFF_AVG_NB帧检测一下同步情况，计数器 */
    double          frame_timer;
    double          frame_last_pts;
    double          frame_last_delay;

    /**
     如果某一帧没有pts信息，就用这个值设置为该帧的pts
     */
    double          video_clock; ///<pts of last decoded frame / predicted pts of next decoded frame
    double          video_current_pts; ///<current displayed pts (different from video_clock if frame fifos are used)
    int64_t         video_current_pts_time; ///<time (av_gettime) at which we updated video_current_pts - used to have running video pts

    AVStream *video_st;
    PacketQueue     videoq;
    VideoPicture    pictq[VIDEO_PICTURE_QUEUE_SIZE];
    int pictq_size, pictq_rindex, pictq_windex;
    SDL_mutex *pictq_mutex;
    SDL_cond *pictq_cond;
    SDL_Thread *parse_tid;
    SDL_Thread *video_tid;

    char            filename[1024];
    int quit;

    AVIOContext *io_context;
    struct SwsContext *sws_ctx;
} VideoState;

enum {
    AV_SYNC_AUDIO_MASTER,
    AV_SYNC_VIDEO_MASTER,
    AV_SYNC_EXTERNAL_MASTER,
};

/* Since we only have one decoding thread, the Big Struct
   can be global in case we need it. */
VideoState *global_video_state;

void packet_queue_init(PacketQueue *q)
{
    memset(q, 0, sizeof(PacketQueue));
    q->mutex = SDL_CreateMutex();
    q->cond = SDL_CreateCond();
}

int packet_queue_put(PacketQueue *q, AVPacket *pkt)
{
    AVPacketList *pkt1;
    if(av_dup_packet(pkt) < 0) {
        return -1;
    }
    pkt1 = av_malloc(sizeof(AVPacketList));
    if (!pkt1)
        return -1;
    pkt1->pkt = *pkt;
    pkt1->next = NULL;

    SDL_LockMutex(q->mutex);

    if (!q->last_pkt)
        q->first_pkt = pkt1;
    else
        q->last_pkt->next = pkt1;
    q->last_pkt = pkt1;
    q->nb_packets++;
    q->size += pkt1->pkt.size;
    SDL_CondSignal(q->cond);

    SDL_UnlockMutex(q->mutex);
    return 0;
}

static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block)
{
    AVPacketList *pkt1;
    int ret;

    SDL_LockMutex(q->mutex);

    for(;;) {

        if(global_video_state->quit) {
            ret = -1;
            break;
        }

        pkt1 = q->first_pkt;
        if (pkt1) {
            q->first_pkt = pkt1->next;
            if (!q->first_pkt)
                q->last_pkt = NULL;
            q->nb_packets--;
            q->size -= pkt1->pkt.size;
            *pkt = pkt1->pkt;
            av_free(pkt1);
            ret = 1;
            break;
        } else if (!block) {
            ret = 0;
            break;
        } else {
            SDL_CondWait(q->cond, q->mutex);
        }
    }

    SDL_UnlockMutex(q->mutex);
    return ret;
}

double get_audio_clock(VideoState *is)
{
    double pts;
    int hw_buf_size, bytes_per_sec, n;

    pts = is->audio_clock; /* maintained in the audio thread */
    hw_buf_size = is->audio_buf_size - is->audio_buf_index;
    bytes_per_sec = 0;
    n = is->audio_st->codec->channels * 2;
    if(is->audio_st) {
        bytes_per_sec = is->audio_st->codec->sample_rate * n;
    }
    if(bytes_per_sec) {
        pts -= (double)hw_buf_size / bytes_per_sec;
    }

    return pts;
}

double get_video_clock(VideoState *is)
{
    double delta;
    delta = (av_gettime() - is->video_current_pts_time) / 1000000.0;
    return is->video_current_pts + delta;
}

double get_external_clock(VideoState *is)
{
    return av_gettime() / 1000000.0;
}

double get_master_clock(VideoState *is)
{
    if(is->av_sync_type == AV_SYNC_VIDEO_MASTER) {
        return get_video_clock(is);
    } else if(is->av_sync_type == AV_SYNC_AUDIO_MASTER) {
        return get_audio_clock(is);
    } else {
        return get_external_clock(is);
    }
}

/* Add or subtract samples to get a better sync, return new audio buffer size */
int synchronize_audio(VideoState *is, short *samples, int samples_size, double pts)
{
    int n;
    double ref_clock;

    //一个采样n个字节, 不理解为什么乘以2，难道是(16bits)
    n = 2 * is->audio_st->codec->channels;

    if(is->av_sync_type == AV_SYNC_AUDIO_MASTER)
        goto funout;

    double diff, avg_diff;
    int wanted_size, min_size, max_size /*, nb_samples */;
    ref_clock = get_master_clock(is);
    diff = get_audio_clock(is) - ref_clock;

    if(diff < AV_NOSYNC_THRESHOLD) {
        // accumulate the diffs
        is->audio_diff_cum = diff + is->audio_diff_avg_coef * is->audio_diff_cum;
        if(is->audio_diff_avg_count < AUDIO_DIFF_AVG_NB) {
            is->audio_diff_avg_count++;
        } else {
            /**
             加权平均，离当前越近的值，权重越大，即影响越大
             */
            avg_diff = is->audio_diff_cum * (1.0 - is->audio_diff_avg_coef);
            if(fabs(avg_diff) >= is->audio_diff_threshold) {
                wanted_size = samples_size + ((int)(diff * is->audio_st->codec->sample_rate) * n);
                min_size = samples_size * ((100 - SAMPLE_CORRECTION_PERCENT_MAX) / 100);
                max_size = samples_size * ((100 + SAMPLE_CORRECTION_PERCENT_MAX) / 100);
                if(wanted_size < min_size) {
                    wanted_size = min_size;
                } else if (wanted_size > max_size) {
                    wanted_size = max_size;
                }
                if(wanted_size < samples_size) {
                    /* remove samples */
                    samples_size = wanted_size;
                } else if(wanted_size > samples_size) {
                    uint8_t *samples_end, *q;
                    int nb;

                    /* add samples by copying final sample*/
                    //nb = (samples_size - wanted_size); //源代码，怀疑这里写错了
                    nb = wanted_size - samples_size;
                    samples_end = (uint8_t *)samples + samples_size - n;
                    q = samples_end + n;
                    while(nb > 0) {
                        memcpy(q, samples_end, n);
                        q += n;
                        nb -= n;
                    }
                    samples_size = wanted_size;
                }
            }
        }
    } else {
        /* difference is TOO big; reset diff stuff */
        is->audio_diff_avg_count = 0;
        is->audio_diff_cum = 0;
    }

funout:
    return samples_size;
}

int audio_decode_frame(VideoState *is, double *pts_ptr)
{
    int len1, data_size = 0, n;
    AVPacket *pkt = &is->audio_pkt;
    double pts;

    for(;;) {
        while(is->audio_pkt_size > 0) {
            int got_frame = 0;
            /**
             一个AVPicture数据包含多个音频帧的编码后信息，一次解码一帧音频帧，故分多次解码
             返回值len1表示，一次解码了len1字节的原始编码音频数据，len1的音频编码数据解码后长度为data_size
             */
            len1 = avcodec_decode_audio4(is->audio_st->codec, &is->audio_frame, &got_frame, pkt);
            if(len1 < 0) {
                /* if error, skip frame */
                is->audio_pkt_size = 0;
                break;
            }
            if (got_frame){
                data_size =
                av_samples_get_buffer_size(NULL, is->audio_st->codec->channels,is->audio_frame.nb_samples,is->audio_st->codec->sample_fmt,1);
                memcpy(is->audio_buf, is->audio_frame.data[0], data_size);
            }
            is->audio_pkt_data += len1;
            is->audio_pkt_size -= len1;
            if(data_size <= 0) {
                /* No data yet, get more frames */
                continue;
            }

            pts = is->audio_clock;
            *pts_ptr = pts;
            n = 2 * is->audio_st->codec->channels;
            /**
             每个音频AVPacket里又有多帧音频AVFrame，计算每个AVFrame的pts值
             */
            is->audio_clock += (double)data_size /
            (double)(n * is->audio_st->codec->sample_rate);

            /* We have data, return it and come back for more later */
            return data_size;
        }
        if(pkt->data)
            av_free_packet(pkt);

        if(is->quit) {
            return -1;
        }

        /* next packet */
        if(packet_queue_get(&is->audioq, pkt, 1) < 0) {
            return -1;
        }

        is->audio_pkt_data = pkt->data;
        is->audio_pkt_size = pkt->size;
        /* if update, update the audio clock w/pts */
        /**
         初始化audio_clock为一个音频AVPacket的pts值
         */
        if(pkt->pts != AV_NOPTS_VALUE) {
            is->audio_clock = av_q2d(is->audio_st->time_base)*pkt->pts;
        }
    }
}

void audio_callback(void *userdata, Uint8 *stream, int len)
{
    VideoState *is = (VideoState *)userdata;
    int len1, audio_size;
    double pts;

    while(len > 0) {
        if(is->audio_buf_index >= is->audio_buf_size) {
            /* We have already sent all our data; get more */
            /**
             audio_decode_frame函数返回一个音频帧数据长度（解码后）
             */
            audio_size = audio_decode_frame(is, &pts);
            if(audio_size < 0) {
                /* If error, output silence */
                is->audio_buf_size = 1024;
                memset(is->audio_buf, 0, is->audio_buf_size);
            } else {
                /**
                 同步音频到其他时钟, 假设同步到视频，假设此时音频比视频快，
                 假设视频播放到第7秒，而音频播放到第10秒, 那就让第10秒的音频连续播放3秒
                 这样当视频播放到第11秒的时候 音频也能刚好播放到第11秒（连续播放的实现是重复最后一个采样）

                 如果音频比视频慢，视频播放到7秒，音频播放到5秒，截断相关采样，让音频接下来6, 7, 8秒要播放的采样
                 在第6秒全部播放完，这样在视频播放的第9秒的时候，音频也刚好播放到第9秒
                 */
                audio_size = synchronize_audio(is, (int16_t *)is->audio_buf, audio_size, pts);
                is->audio_buf_size = audio_size;
            }
            is->audio_buf_index = 0;
        }
        len1 = is->audio_buf_size - is->audio_buf_index;
        if(len1 > len)
            len1 = len;
        memcpy(stream, (uint8_t *)is->audio_buf + is->audio_buf_index, len1);
        len -= len1;
        stream += len1;
        is->audio_buf_index += len1;
    }
}

static Uint32 sdl_refresh_timer_cb(Uint32 interval, void *opaque)
{
    SDL_Event event;
    event.type = FF_REFRESH_EVENT;
    event.user.data1 = opaque;
    SDL_PushEvent(&event);
    return 0; /* 0 means stop timer */
}

/* schedule a video refresh in 'delay' ms */
static void schedule_refresh(VideoState *is, int delay)
{
    SDL_AddTimer(delay, sdl_refresh_timer_cb, is);
}

void video_display(VideoState *is)
{
    SDL_Rect rect = {0};
    VideoPicture *vp;
    //AVPicture pict;
    float aspect_ratio;
    int w, h, x, y;
    //int i;
    vp = &is->pictq[is->pictq_rindex];
    if(vp->bmp) {
        if(is->video_st->codec->sample_aspect_ratio.num == 0) {
            aspect_ratio = 0;
        } else {
            aspect_ratio = av_q2d(is->video_st->codec->sample_aspect_ratio) * is->video_st->codec->width / is->video_st->codec->height;
        }

        if(aspect_ratio <= 0.0) {
            aspect_ratio = (float)is->video_st->codec->width / (float)is->video_st->codec->height;
        }

        //h = screen->h;
        //w = ((int)rint(h * aspect_ratio)) & -3;
        //if(w > screen->w) {
        //    w = screen->w;
        //    h = ((int)rint(w / aspect_ratio)) & -3;
        //}
        //x = (screen->w - w) / 2;
        //y = (screen->h - h) / 2;

        rect.x = 0;
        rect.y = 0;
        //rect.w = w;
        //rect.h = h;
        rect.w = vp->width;
        rect.h = vp->height;

        SDL_UpdateTexture(vp->bmp, &rect, vp->pFrameYUV->data[0], vp->pFrameYUV->linesize[0] );
        SDL_RenderClear( renderer );
        SDL_RenderCopy( renderer, vp->bmp, &rect, &rect );
        SDL_RenderPresent( renderer );

        free(vp->bufpoint);
        av_free(vp->pFrameYUV);
        // SDL_DisplayYUVOverlay(vp->bmp, &rect);
    }
}

void video_refresh_timer(void *userdata)
{
    VideoState *is = (VideoState *)userdata;

    VideoPicture *vp;
    double actual_delay, delay, sync_threshold, ref_clock, diff;

    if(is->video_st) {
        if(is->pictq_size == 0) {
            schedule_refresh(is, 1);
        } else {
            vp = &is->pictq[is->pictq_rindex];

            is->video_current_pts = vp->pts;
            is->video_current_pts_time = av_gettime();
            /**
             两个视频帧的pts差值，在(0s, 1.0s)这个区间内，如果帧率是25fps, 那么这个值应该在40ms,即0.04s附近波动
             如果不在上述区间，则把这个delta pts设置成与上一个相同
             */
            delay = vp->pts - is->frame_last_pts; /* the pts from last time */
            if(delay <= 0 || delay >= 1.0) {
                /* if incorrect delay, use previous one */
                delay = is->frame_last_delay;
            }
            /* save for next time */
            is->frame_last_delay = delay;
            is->frame_last_pts = vp->pts;

            /* update delay to sync to audio if not master source */
            /**
             视频跟自己同步就不需要改变delay, 如果是同步到音频，或者wallclock就需要判断是否在同步阀值内
             是否需要改变delay的值
             */
            if(is->av_sync_type != AV_SYNC_VIDEO_MASTER) {
                ref_clock = get_master_clock(is);
                diff = vp->pts - ref_clock;

                /* Skip or repeat the frame. Take delay into account
                   FFPlay still doesn't "know if this is the best guess." */
                sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD;
                if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
                    if(diff <= -sync_threshold) {
                        delay = 0;
                    } else if(diff >= sync_threshold) {
                        delay = 2 * delay;
                    }
                }
            }
            is->frame_timer += delay;

            /* computer the REAL delay */
            /**
             在stream_component_open处理视频的时候有初始化一个值frame_timer, is->frame_timer = (double)av_gettime() / 1000000.0;
             我们把这个时间设为time_a, 这个认为是视频最开始的时间，现在假设最开始的情况是开始播放第一帧视频，delay秒后播放第一帧视频
             即要在wallclock的 time_a + delay秒开始播放第一帧，设这个时间为time_c, 当前时间是time_b. 时间关系如下所示
             time_a ------ time_b ------------------------------------------------ time_c
             如果我们在time_b还延迟delay秒播放第一帧就不准确了, 所以需要av_gettime函数来校正delay时间获取actual_delay
             */
            actual_delay = is->frame_timer - (av_gettime() / 1000000.0);
            if(actual_delay < 0.010) {
                /* Really it should skip the picture instead */
                actual_delay = 0.010;
            }
            schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));

            /* show the */
            video_display(is);

            /* update queue for next */
            if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) {
                is->pictq_rindex = 0;
            }
            SDL_LockMutex(is->pictq_mutex);
            is->pictq_size--;
            SDL_CondSignal(is->pictq_cond);
            SDL_UnlockMutex(is->pictq_mutex);
        }
    } else {
        schedule_refresh(is, 100);
    }
}

void alloc_picture(void *userdata)
{
    VideoState *is = (VideoState *)userdata;
    VideoPicture *vp;

    vp = &is->pictq[is->pictq_windex];
    if(vp->bmp) {
        // we already have one make another, bigger/smaller
        SDL_DestroyTexture(vp->bmp);
    }

    // Allocate a place to put our YUV image on that screen
    vp->bmp = SDL_CreateTexture(renderer, SDL_PIXELFORMAT_YV12, SDL_TEXTUREACCESS_STREAMING, is->video_st->codec->width, is->video_st->codec->height);
    vp->width = is->video_st->codec->width;
    vp->height = is->video_st->codec->height;

    SDL_LockMutex(is->pictq_mutex);
    vp->allocated = 1;
    SDL_CondSignal(is->pictq_cond);
    SDL_UnlockMutex(is->pictq_mutex);
}

int queue_picture(VideoState *is, AVFrame *pFrame, double pts)
{
    VideoPicture *vp;
    SDL_Rect rect;

    /* wait until we have space for a new pic */
    SDL_LockMutex(is->pictq_mutex);
    while(is->pictq_size >= VIDEO_PICTURE_QUEUE_SIZE && !is->quit) {
        SDL_CondWait(is->pictq_cond, is->pictq_mutex);
    }
    SDL_UnlockMutex(is->pictq_mutex);

    if(is->quit)
        return -1;

    // windex is set to 0 initially
    vp = &is->pictq[is->pictq_windex];

    /* allocate or resize the */
    if(!vp->bmp || vp->width != is->video_st->codec->width || vp->height != is->video_st->codec->height) {
        SDL_Event event;

        vp->allocated = 0;
        /* we have to do it in the main thread */
        event.type = FF_ALLOC_EVENT;
        event.user.data1 = is;
        SDL_PushEvent(&event);

        /* wait until we have a picture allocated */
        SDL_LockMutex(is->pictq_mutex);
        while(!vp->allocated && !is->quit) {
            SDL_CondWait(is->pictq_cond, is->pictq_mutex);
        }
        SDL_UnlockMutex(is->pictq_mutex);
        if(is->quit) {
            return -1;
        }
    }
    /* We have a place to put our picture on the queue */
    /* If we are skipping a frame, do we set this to null but still return vp->allocated = 1? */

    if(vp->bmp) {
        vp->pFrameYUV = avcodec_alloc_frame();
        // SDL_LockYUVOverlay(vp->bmp);
        int numBytes = avpicture_get_size(PIX_FMT_YUV420P, is->video_st->codec->width, is->video_st->codec->height);
        uint8_t* buffer = (uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
        vp->bufpoint = buffer;
        avpicture_fill((AVPicture *)vp->pFrameYUV, buffer, PIX_FMT_YUV420P, is->video_st->codec->width, is->video_st->codec->height);
        /* point pict at the queue */

        // Convert the image into YUV format that SDL uses
        sws_scale(is->sws_ctx,(uint8_t const * const *)pFrame->data,pFrame->linesize,0, is->video_st->codec->height, vp->pFrameYUV->data, vp->pFrameYUV->linesize);

        // SDL_UnlockYUVOverlay(vp->bmp);
        vp->pts = pts;

        /* now we inform our display thread that we have a pic ready */
        if(++is->pictq_windex == VIDEO_PICTURE_QUEUE_SIZE) {
            is->pictq_windex = 0;
        }

        SDL_LockMutex(is->pictq_mutex);
        is->pictq_size++;
        SDL_UnlockMutex(is->pictq_mutex);
    }

    return 0;
}

double synchronize_video(VideoState *is, AVFrame *src_frame, double pts)
{
    double frame_delay;

    //pts of last decoded frame
    if(pts != 0) {
        /* if we have pts, set video clock to it */
        is->video_clock = pts;
    } else {
        /* if we aren't given a pts, set it to the clock */
        pts = is->video_clock;
    }

    //predicted pts of next decoded frame
    /* update the video clock */
    frame_delay = av_q2d(is->video_st->codec->time_base);    // 1 / fps
    /* if we are repeating a frame, adjust clock accordingly */
    frame_delay += src_frame->repeat_pict * (frame_delay * 0.5);    // why 0.5? Do not understand this
    is->video_clock += frame_delay;

    return pts;
}

uint64_t global_video_pkt_pts = AV_NOPTS_VALUE;
int video_thread(void *arg)
{
    VideoState *is = (VideoState *)arg;
    AVPacket pkt1, *packet = &pkt1;
    int frameFinished;
    AVFrame *pFrame;
    double pts;

    pFrame = av_frame_alloc();

    for(;;) {
        if(packet_queue_get(&is->videoq, packet, 1) < 0) {
            // means we quit getting packets
            break;
        }
        pts = 0;

        // Save global pts to be stored in pFrame in first call
        global_video_pkt_pts = packet->pts;
        // Decode video frame
        /**
         The pts of a frame equal to packet's pts which is the first packet of this frame.
         But we do not know which packet is the first packet of a frame.
         Whenever a packet starts a frame, the avcodec_decode_video() will call a function to allocate a buffer in our frame.
         And of course, ffmpeg allows us to redefine what that allocation function is. (our_get_buffer)
         So we'll make a new function that saves the pts of the packet.
        */
        avcodec_decode_video2(is->video_st->codec, pFrame, &frameFinished, packet);

        if(packet->dts == AV_NOPTS_VALUE && pFrame->opaque && *(uint64_t*)pFrame->opaque != AV_NOPTS_VALUE) {
            pts = *(uint64_t *)pFrame->opaque;
        } else if(packet->dts != AV_NOPTS_VALUE) {
            pts = packet->dts;
        } else {
            pts = 0;
        }
        /**
         pts value is a timestamp that corresponds to a measurement of time in that stream's time_base unit.
         For example, if a stream has time_base unit = 25hz/s ,the a PTS of 50 means 2s
         Just like pts's Frequency equal to 90khz/s ,and wallclock's Frequency equal to 1hz/s
         If we want to transform a pts to wallclock we can calculate it like this: wallclock = pts * (1/90000)
         */
        pts *= av_q2d(is->video_st->time_base);

        av_free_packet(packet);

        // Did we get a video frame?
        if(frameFinished) {
            pts = synchronize_video(is, pFrame, pts);
            if(queue_picture(is, pFrame, pts) < 0) {
                break;
            }
        }
    }

    av_free(pFrame);
    return 0;
}

/* These are called whenever we allocate a frame
 * buffer. We use this to store the global_pts in
 * a frame at the time it is allocated.
 */
int our_get_buffer(struct AVCodecContext *c, AVFrame *pic, int flags)
{
//    int ret = avcodec_default_get_buffer(c, pic);
    int ret = avcodec_default_get_buffer2(c, pic, flags);
    uint64_t *pts = av_malloc(sizeof(uint64_t));

    *pts = global_video_pkt_pts;
    pic->opaque = pts;

    return ret;
}

void our_release_buffer(struct AVCodecContext *c, AVFrame *pic)
{
    if(pic) av_freep(&pic->opaque);

//    avcodec_default_release_buffer(c, pic);
    if(!(c->codec_type == AVMEDIA_TYPE_VIDEO))
        abort();
    av_frame_unref(pic);
}

int stream_component_open(VideoState *is, int stream_index)
{
    AVFormatContext *pFormatCtx = is->pFormatCtx;
    AVCodecContext *codecCtx = NULL;
    AVCodec *codec = NULL;
    AVDictionary *optionsDict = NULL;
    SDL_AudioSpec wanted_spec, spec;

    if(stream_index < 0 || stream_index >= pFormatCtx->nb_streams) {
        return -1;
    }

    // Get a pointer to the codec context for the video stream
    codecCtx = pFormatCtx->streams[stream_index]->codec;

    if(codecCtx->codec_type == AVMEDIA_TYPE_AUDIO) {
        // Set audio settings from codec info
        wanted_spec.freq = codecCtx->sample_rate;
        wanted_spec.format = AUDIO_S16SYS;
        wanted_spec.channels = codecCtx->channels;
        wanted_spec.silence = 0;
        wanted_spec.samples = SDL_AUDIO_BUFFER_SIZE;
        wanted_spec.callback = audio_callback;
        wanted_spec.userdata = is;

        //open a audio device
        if(SDL_OpenAudio(&wanted_spec, &spec) < 0) {
            fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError());
            return -1;
        }
        is->audio_hw_buf_size = spec.size;
    }
    codec = avcodec_find_decoder(codecCtx->codec_id);
    if(!codec || (avcodec_open2(codecCtx, codec, &optionsDict) < 0)) {
        fprintf(stderr, "Unsupported codec!\n");
        return -1;
    }

    switch(codecCtx->codec_type) {
        case AVMEDIA_TYPE_AUDIO:
            is->audioStream = stream_index;
            is->audio_st = pFormatCtx->streams[stream_index];
            is->audio_buf_size = 0;
            is->audio_buf_index = 0;
            /* averaging filter for audio sync */
            is->audio_diff_avg_coef = exp(log(0.01 / AUDIO_DIFF_AVG_NB));    /** 0.0005 */
            is->audio_diff_avg_count = 0;
            /* Correct audio only if larger error than this */
            is->audio_diff_threshold = 2.0 * SDL_AUDIO_BUFFER_SIZE / codecCtx->sample_rate;
            memset(&is->audio_pkt, 0, sizeof(is->audio_pkt));
            packet_queue_init(&is->audioq);
            //play the audio
            SDL_PauseAudio(0);
            break;
        case AVMEDIA_TYPE_VIDEO:
            is->videoStream = stream_index;
            is->video_st = pFormatCtx->streams[stream_index];
            //initialize the frame timer and the initial previous frame delay(40ms)
            is->frame_timer = (double)av_gettime() / 1000000.0;
            is->frame_last_delay = 40e-3;
            is->video_current_pts_time = av_gettime();
            packet_queue_init(&is->videoq);
            is->sws_ctx = sws_getContext(is->video_st->codec->width,is->video_st->codec->height,is->video_st->codec->pix_fmt,is->video_st->codec->width,
                            is->video_st->codec->height,PIX_FMT_YUV420P, SWS_BILINEAR, NULL, NULL, NULL);
            codecCtx->get_buffer2 = our_get_buffer;
            codecCtx->release_buffer = our_release_buffer;
            is->video_tid = SDL_CreateThread(video_thread, "video_tid", is);
            break;
        default:
            break;
    }

    return 0;
}

int decode_interrupt_cb(void *opaque)
{
    return (global_video_state && global_video_state->quit);
}

int decode_thread(void *arg)
{
    VideoState *is = (VideoState *)arg;
    AVFormatContext *pFormatCtx = NULL;
    AVPacket pkt1, *packet = &pkt1;

    int video_index = -1;
    int audio_index = -1;
    int i;

    AVDictionary *io_dict = NULL;
    AVIOInterruptCB callback;

    is->videoStream=-1;
    is->audioStream=-1;

    global_video_state = is;

    // will interrupt blocking functions if we
    callback.callback = decode_interrupt_cb;
    callback.opaque = is;

    /**
      不理解这个io_context到底怎么回事？
      只在这里初始化，并未关联到pFormatCtx
     */
    if (avio_open2(&is->io_context, is->filename, 0, &callback, &io_dict)){
        fprintf(stderr, "Unable to open I/O for %s\n", is->filename);
        return -1;
    }

    // Open video file
    if(avformat_open_input(&pFormatCtx, is->filename, NULL, NULL)!=0)
        return -1; // Couldn't open file

    is->pFormatCtx = pFormatCtx;

    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0)
        return -1; // Couldn't find stream information

    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, is->filename, 0);

    // Find the first video stream
    for(i=0; i<pFormatCtx->nb_streams; i++) {
        if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO && video_index < 0) {
            video_index=i;
        }
        if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO && audio_index < 0) {
            audio_index=i;
        }
    }
    if(audio_index >= 0) {
        stream_component_open(is, audio_index);
    }
    if(video_index >= 0) {
        stream_component_open(is, video_index);
    }

    if(is->videoStream < 0 || is->audioStream < 0) {
        fprintf(stderr, "%s: could not open codecs\n", is->filename);
        goto fail;
    }

    // main decode loop
    for(;;) {
        if(is->quit) {
            break;
        }
        // seek stuff goes here
        if(is->audioq.size > MAX_AUDIOQ_SIZE || is->videoq.size > MAX_VIDEOQ_SIZE) {
            SDL_Delay(10);
            continue;
        }
        if(av_read_frame(is->pFormatCtx, packet) < 0) {
            is->quit = global_readframe_cntout++ < 8? 0:1;
            SDL_Delay(200);
            /*
            if(is->pFormatCtx->pb->error == 0) {
                SDL_Delay(100); // no error; wait for user input
                continue;
            } else {
                break;
            }
            */
        }

        // Is this a packet from the video stream?
        if(packet->stream_index == is->videoStream) {
            packet_queue_put(&is->videoq, packet);
        } else if(packet->stream_index == is->audioStream) {
            packet_queue_put(&is->audioq, packet);
        } else {
            av_free_packet(packet);
        }
    }

    /* all done - wait for it */
    while(!is->quit) {
        SDL_Delay(100);
    }

fail:
    if(1){
        SDL_Event event;
        event.type = FF_QUIT_EVENT;
        event.user.data1 = is;
        SDL_PushEvent(&event);
    }
    return 0;
}

int main(int argc, char *argv[])
{
    if(argc < 2) {
        fprintf(stderr, "Usage: test \n");
        return -1;
    }

    SDL_Event       event = {0};
    VideoState *is;
      is = av_mallocz(sizeof(VideoState));
    if(!is) return -1;

    if(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) {
        fprintf(stderr, "Could not initialize SDL - %s\n", SDL_GetError());
        return -1;
    }

    // Register all formats and codecs
    av_register_all();

    // Make a screen to put our video
      screen = SDL_CreateWindow("Window", SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, PRE_WIDTH, PRE_HEIGHT, SDL_WINDOW_FULLSCREEN_DESKTOP);
    renderer = SDL_CreateRenderer(screen, -1, 0);
    if(!screen) {
        fprintf(stderr, "SDL: could not set video mode - exiting (%s)\n", SDL_GetError());
        av_free(is);
        return -1;
    }

    av_strlcpy(is->filename, argv[1], sizeof(is->filename));
    is->pictq_mutex = SDL_CreateMutex();
    is->pictq_cond = SDL_CreateCond();

    schedule_refresh(is, 40);    //this line aim to send a FF_REFRESH_EVENT signal to main function 40ms later

    is->av_sync_type = DEFAULT_AV_SYNC_TYPE;
    is->parse_tid = SDL_CreateThread(decode_thread, "parse_tid", is);
    if(!is->parse_tid) {
        av_free(is);
        return -1;
    }

    for(;;) {
        SDL_WaitEvent(&event);
        switch(event.type) {
            case FF_QUIT_EVENT:
            case SDL_QUIT:
                is->quit = 1;
                /*
                * If the video has finished playing, then both the picture and
                * audio queues are waiting for more data. Make them stop
                * waiting and terminate normally.
                */
                SDL_CondSignal(is->audioq.cond);
                SDL_CondSignal(is->videoq.cond);
                SDL_Quit();
                av_free(is);
                return 0;
            case FF_ALLOC_EVENT:
                alloc_picture(event.user.data1);
                break;
            case FF_REFRESH_EVENT:
                video_refresh_timer(event.user.data1);
                break;
            default:
                break;
        }
    }

    av_free(is);
    return 0;
}