Chinaunix首页 | 论坛 | 博客
  • 博客访问: 335000
  • 博文数量: 41
  • 博客积分: 0
  • 博客等级: 民兵
  • 技术积分: 476
  • 用 户 组: 普通用户
  • 注册时间: 2016-09-01 19:08
个人简介

Android/Linux/音频/驱动

文章分类

全部博文(41)

文章存档

2017年(21)

2016年(20)

我的朋友

分类: Android平台

2017-02-24 11:05:11

Android 7.0 Audio的Resample过程详解

Qidi 2017.02.23 (Markdown & Haroopad)


【前言】

处理过音频文件的工程师都知道音频数据存在采样率(Sample Rate)这个指标。在位深度(Bit Depth)一定的情况下,采样率越高,理论上来说播放出来的声音就越细腻,录制的声音也就越保真,反之亦然。

但在较早的Android系统版本上,不管音频文件原来的采样率几何,统统都被重采样(Resample)到44.1KHz进行播放,录制的时候则是被固定为8KHz进行采样。尽管这样的处理方式被广大音质爱好者所诟病,但在当时它确实是一种实现设备兼容的有效方法。

作为Android Audio BSP工程师,有必要了解系统实现Resample的过程。现在Android系统已经发布到了7.0版本,一起看看在最新的版本上这个Resample的过程是怎样实现的吧。

【背景知识】

我们知道在Android系统中,当应用层APP播放一个音频文件时,Framework层的AudioPolicyService(APS)会接收上层APP传递来的音频参数(例如格式、声道、采样率等),并调用AudioFlinger的createTrack()方法对应创建1个Track,再调用openOutput()方法来打开1个outputStream,然后使用这个outputStream来创建相应的Playback线程(依据应用场景可以是OffloadThread、DirectOutputThread、MixerThread),最终在Playback线程中匹配之前创建的Track,开始自APP至HAL的数据传输。

【Resample过程分析】

那么我们对Android Audio Resample过程的分析就从AudioFlinger开始。在AudioFlinger::openOutput()中可以看到,在Playback线程被成功创建之后,即被加入到mPlaybackThreads向量中进行管理了。具体代码如下:

  1. sp<AudioFlinger::PlaybackThread> AudioFlinger::openOutput_l(audio_module_handle_t module,
  2.                                                             audio_io_handle_t *output,
  3.                                                             audio_config_t *config,
  4.                                                             audio_devices_t devices,
  5.                                                             const String8& address,
  6.                                                             audio_output_flags_t flags)
  7. {
  8.     ......
  9.     
  10.     AudioStreamOut *outputStream = NULL;
  11.     status_t status = outHwDev->openOutputStream( // 打开1个outputStream
  12.             &outputStream,
  13.             *output,
  14.             devices,
  15.             flags,
  16.             config,
  17.             address.string());

  18.     mHardwareStatus = AUDIO_HW_IDLE;

  19.     if (status == NO_ERROR) {

  20.         PlaybackThread *thread;
  21.         if (flags & AUDIO_OUTPUT_FLAG_COMPRESS_OFFLOAD) {
  22.             thread = new OffloadThread(this, outputStream, *output, devices, mSystemReady);
  23.             ALOGV("openOutput_l() created offload output: ID %d thread %p", *output, thread);
  24.         } else if ((flags & AUDIO_OUTPUT_FLAG_DIRECT)
  25.                 || !isValidPcmSinkFormat(config->format)
  26.                 || !isValidPcmSinkChannelMask(config->channel_mask)) {
  27.             thread = new DirectOutputThread(this, outputStream, *output, devices, mSystemReady);
  28.             ALOGV("openOutput_l() created direct output: ID %d thread %p", *output, thread);
  29.         } else {
  30.             thread = new MixerThread(this, outputStream, *output, devices, mSystemReady); /* 默认情况下,创建MixerThread类型的Playback线程 */
  31.             ALOGV("openOutput_l() created mixer output: ID %d thread %p", *output, thread);
  32.         }
  33.         mPlaybackThreads.add(*output, thread); /* 将新创建的线程加入向量 */
  34.         return thread;
  35.     }

  36.     return 0;
  37. }

随后Playback线程运行,对应的AudioFlinger::Playback::threadLoop()方法被执行,在该方法中调用了prepareTracks_l()函数。这个函数实际上是对应于AudioFlinger::MixerThread::prepareTracks_l()这个函数。threadLoop()函数代码细节如下:

  1. bool AudioFlinger::PlaybackThread::threadLoop()
  2. {
  3.     Vector< sp<Track> > tracksToRemove;
  4.     ......

  5.     while (!exitPending())
  6.     {
  7.         cpuStats.sample(myName);
  8.         Vector< sp<EffectChain> > effectChains;
  9.         { // scope for mLock
  10.             Mutex::Autolock _l(mLock);
  11.             processConfigEvents_l();
  12.             ......
  13.             saveOutputTracks();
  14.             ......
  15.             if ((!mActiveTracks.size() && systemTime() > mStandbyTimeNs) ||
  16.                                    isSuspended()) {
  17.                 // put audio hardware into standby after short delay
  18.                 if (shouldStandby_l()) {

  19.                     threadLoop_standby();

  20.                     mStandby = true;
  21.                 }
  22.                 ......
  23.             }
  24.             // mMixerStatusIgnoringFastTracks is also updated internally
  25.             mMixerStatus = prepareTracks_l(&tracksToRemove); /* 调用prepareTracks_l(),为Playback线程匹配已注册的Track */
  26.             ......
  27.             // prevent any changes in effect chain list and in each effect chain
  28.             // during mixing and effect process as the audio buffers could be deleted
  29.             // or modified if an effect is created or deleted
  30.             lockEffectChains_l(effectChains);
  31.         } // mLock scope ends
  32.         ......
  33.         // enable changes in effect chain
  34.         unlockEffectChains(effectChains);
  35.         // Finally let go of removed track(s), without the lock held
  36.         // since we can't guarantee the destructors won't acquire that
  37.         // same lock. This will also mutate and push a new fast mixer state.
  38.         threadLoop_removeTracks(tracksToRemove);
  39.         tracksToRemove.clear();
  40.         // FIXME I don't understand the need for this here;
  41.         // it was in the original code but maybe the
  42.         // assignment in saveOutputTracks() makes this unnecessary?
  43.         clearOutputTracks();
  44.         // Effect chains will be actually deleted here if they were removed from
  45.         // mEffectChains list during mixing or effects processing
  46.         effectChains.clear();
  47.         // FIXME Note that the above .clear() is no longer necessary since effectChains
  48.         // is now local to this block, but will keep it for now (at least until merge done).
  49.     }
  50.     threadLoop_exit();
  51.     if (!mStandby) {
  52.         threadLoop_standby();
  53.         mStandby = true;
  54.     }
  55.     ......
  56.     return false;
  57. }

Resample的过程就发生在prepareTracks_l()函数中,所以我们来好好阅读一下。在该函数中,通过一个for循环遍历所有处于active状态的track。每一次循环中,都要进行如下2步操作: 

1. 通过reqSampleRate = track->mAudioTrackServerProxy->getSampleRate()来获取硬件设备所支持的采样率; 
2. 之后调用mAudioMixer->setParameter(name, AudioMixer::RESAMPLE, AudioMixer::SAMPLE_RATE, (void*)(uintptr_t)reqSampleRate),通过对比音频文件采样率和音频设备支持的采样率,判断是否创建新的Resampler对象,或者从已有的Resampler对象列表中返回1个;

prepareTracks_l()函数代码细节如下:

  1. AudioFlinger::PlaybackThread::mixer_state AudioFlinger::MixerThread::prepareTracks_l(
  2.         Vector< sp<Track> > *tracksToRemove)
  3. {
  4.     ......
  5.     // find out which tracks need to be processed
  6.     size_t count = mActiveTracks.size(); /* 获取处于active状态的track的数量 */
  7.     ......
  8.     for (size_t i=0 ; i<count ; i++) {
  9.         const sp<Track> t = mActiveTracks[i].promote();
  10.         if (t == 0) {
  11.             continue;
  12.         }
  13.         // this const just means the local variable doesn't change
  14.         Track* const track = t.get(); /* 获取对应的track */
  15.         ......
  16.         audio_track_cblk_t* cblk = track->cblk();

  17.         // The first time a track is added we wait
  18.         // for all its buffers to be filled before processing it
  19.         int name = track->name();
  20.         ......
  21.         if ((framesReady >= minFrames) && track->isReady() &&
  22.                 !track->isPaused() && !track->isTerminated())
  23.         {
  24.             ......
  25.             int param = AudioMixer::VOLUME;
  26.             if (track->mFillingUpStatus == Track::FS_FILLED) {
  27.                 // no ramp for the first volume setting
  28.                 track->mFillingUpStatus = Track::FS_ACTIVE;
  29.                 if (track->mState == TrackBase::RESUMING) {
  30.                     track->mState = TrackBase::ACTIVE;
  31.                     param = AudioMixer::RAMP_VOLUME;
  32.                 }
  33.                 mAudioMixer->setParameter(name, AudioMixer::RESAMPLE, AudioMixer::RESET, NULL);
  34.             // FIXME should not make a decision based on mServer
  35.             } else if (cblk->mServer != 0) {
  36.                 // If the track is stopped before the first frame was mixed,
  37.                 // do not apply ramp
  38.                 param = AudioMixer::RAMP_VOLUME;
  39.             }

  40.             // compute volume for this track
  41.             ......
  42.             // Delegate volume control to effect in track effect chain if needed
  43.             ......

  44.             // XXX: these things DON'T need to be done each time
  45.             mAudioMixer->setBufferProvider(name, track);
  46.             mAudioMixer->enable(name);

  47.             mAudioMixer->setParameter(name, param, AudioMixer::VOLUME0, &vlf); /* 设置左声道音量 */
  48.             mAudioMixer->setParameter(name, param, AudioMixer::VOLUME1, &vrf); /* 设置右声道音量 */
  49.             mAudioMixer->setParameter(name, param, AudioMixer::AUXLEVEL, &vaf); /* 设置辅助声道音量 */
  50.             mAudioMixer->setParameter(
  51.                 name,
  52.                 AudioMixer::TRACK,
  53.                 AudioMixer::FORMAT, (void *)track->format()); /* 设置音频数据格式 */
  54.             mAudioMixer->setParameter(
  55.                 name,
  56.                 AudioMixer::TRACK,
  57.                 AudioMixer::CHANNEL_MASK, (void *)(uintptr_t)track->channelMask()); /* 设置音频声道数 */
  58.             mAudioMixer->setParameter(
  59.                 name,
  60.                 AudioMixer::TRACK,
  61.                 AudioMixer::MIXER_CHANNEL_MASK, (void *)(uintptr_t)mChannelMask);
  62.             // limit track sample rate to 2 x output sample rate, which changes at re-configuration
  63.             uint32_t maxSampleRate = mSampleRate * AUDIO_RESAMPLER_DOWN_RATIO_MAX;
  64.             uint32_t reqSampleRate = track->mAudioTrackServerProxy->getSampleRate(); /* 获取音频设备所支持的采样率 */
  65.             if (reqSampleRate == 0) {
  66.                 reqSampleRate = mSampleRate;
  67.             } else if (reqSampleRate > maxSampleRate) {
  68.                 reqSampleRate = maxSampleRate;
  69.             }
  70.             mAudioMixer->setParameter(
  71.                 name,
  72.                 AudioMixer::RESAMPLE,
  73.                 AudioMixer::SAMPLE_RATE, /* 设置音频采样率(必要时会进行重采样) */
  74.                 (void *)(uintptr_t)reqSampleRate);

  75.             AudioPlaybackRate playbackRate = track->mAudioTrackServerProxy->getPlaybackRate();
  76.             mAudioMixer->setParameter(
  77.                 name,
  78.                 AudioMixer::TIMESTRETCH,
  79.                 AudioMixer::PLAYBACK_RATE, /* 设置播放码率 */
  80.                 &playbackRate);

  81.             /*
  82.              * Select the appropriate output buffer for the track.
  83.              *
  84.              * Tracks with effects go into their own effects chain buffer
  85.              * and from there into either mEffectBuffer or mSinkBuffer.
  86.              *
  87.              * Other tracks can use mMixerBuffer for higher precision
  88.              * channel accumulation. If this buffer is enabled
  89.              * (mMixerBufferEnabled true), then selected tracks will accumulate
  90.              * into it.
  91.              *
  92.              */
  93.             if (mMixerBufferEnabled
  94.                     && (track->mainBuffer() == mSinkBuffer
  95.                             || track->mainBuffer() == mMixerBuffer)) {
  96.                 mAudioMixer->setParameter(
  97.                         name,
  98.                         AudioMixer::TRACK,
  99.                         AudioMixer::MIXER_FORMAT, (void *)mMixerBufferFormat); /* 设置缓冲区数据格式 */
  100.                 mAudioMixer->setParameter(
  101.                         name,
  102.                         AudioMixer::TRACK,
  103.                         AudioMixer::MAIN_BUFFER, (void *)mMixerBuffer); /* 分配主缓冲区 */
  104.                 // TODO: override track->mainBuffer()?
  105.                 mMixerBufferValid = true;
  106.             } else {
  107.                 ......
  108.             }
  109.             mAudioMixer->setParameter(
  110.                 name,
  111.                 AudioMixer::TRACK,
  112.                 AudioMixer::AUX_BUFFER, (void *)track->auxBuffer()); /* 分配副缓冲区 */

  113.             // reset retry count
  114.             track->mRetryCount = kMaxTrackRetries;

  115.             // If one track is ready, set the mixer ready if:
  116.             // - the mixer was not ready during previous round OR
  117.             // - no other track is not ready
  118.             if (mMixerStatusIgnoringFastTracks != MIXER_TRACKS_READY ||
  119.                     mixerStatus != MIXER_TRACKS_ENABLED) {
  120.                 mixerStatus = MIXER_TRACKS_READY;
  121.             }
  122.         } else {
  123.             // 出现underrun,以及相应处理操作
  124.             ......
  125.         }
  126.     }

  127.     // Push the new FastMixer state if necessary
  128.     ......
  129.     // Now perform the deferred reset on fast tracks that have stopped
  130.     ......
  131.     // remove all the tracks that need to be...
  132.     removeTracks_l(*tracksToRemove);
  133.     ......
  134.     // sink or mix buffer must be cleared if all tracks are connected to an
  135.     // effect chain as in this case the mixer will not write to the sink or mix buffer
  136.     // and track effects will accumulate into it
  137.     ......
  138.     // if any fast tracks, then status is ready
  139.     ......
  140.     return mixerStatus;
  141. }

在确认要使用的Resampler对象存在后,调用invalidateState(1 << name)使设置生效,开始执行重采样。invalidateState()函数会调用AudioMixer::process_validate(),在该函数中首先通过语句t.hook = getTrackHook(TRACKTYPE_RESAMPLE, t.mMixerChannelCount, t.mMixerInFormat, t.mMixerFormat);获取执行重采样操作的函数,随后通过state->hook = process_resampling;中的t.hook(&t, outTemp, numFrames, state->resampleTemp, aux)语句进行调用。 

setParameter()函数代码如下:

  1. void AudioMixer::setParameter(int name, int target, int param, void *value)
  2. {
  3.     ......
  4.     int valueInt = static_cast<int>(reinterpret_cast<uintptr_t>(value));
  5.     int32_t *valueBuf = reinterpret_cast<int32_t*>(value);
  6.     switch (target) {
  7.     ......
  8.     case RESAMPLE:
  9.         switch (param) {
  10.         case SAMPLE_RATE:
  11.             ALOG_ASSERT(valueInt > 0, "bad sample rate %d", valueInt);
  12.             if (track.setResampler(uint32_t(valueInt), mSampleRate)) { /* 新建或查找1个Resampler对象 */
  13.                 ALOGV("setParameter(RESAMPLE, SAMPLE_RATE, %u)",
  14.                         uint32_t(valueInt));
  15.                 invalidateState(1 << name); /* 使设置生效,调用重采样的后续处理函数 */
  16.             }
  17.             break;
  18.         case RESET:
  19.             track.resetResampler();
  20.             invalidateState(1 << name);
  21.             break;
  22.         case REMOVE:
  23.             delete track.resampler;
  24.             track.resampler = NULL;
  25.             track.sampleRate = mSampleRate;
  26.             invalidateState(1 << name);
  27.             break;
  28.         default:
  29.             LOG_ALWAYS_FATAL("setParameter resample: bad param %d", param);
  30.         }
  31.         break;
  32.     }
  33. }

invalidateState()函数代码如下:

  1. void AudioMixer::invalidateState(uint32_t mask)
  2. {
  3.     if (mask != 0) {
  4.         mState.needsChanged |= mask;
  5.         mState.hook = process__validate;
  6.     }
  7. }
process__validate()函数代码如下:
  1. void AudioMixer::process__validate(state_t* state)
  2. {
  3.     ......
  4.     uint32_t en = state->enabledTracks;
  5.     while (en) {
  6.         ......
  7.         if (n & NEEDS_MUTE) {
  8.             ......
  9.         } else {
  10.             ......
  11.             if (n & NEEDS_RESAMPLE) {
  12.                 all16BitsStereoNoResample = false;
  13.                 resampling = true;
  14.                 t.hook = getTrackHook(TRACKTYPE_RESAMPLE, t.mMixerChannelCount,
  15.                         t.mMixerInFormat, t.mMixerFormat); /* 获取track__genericResample()函数 */
  16.                 ALOGV_IF((n & NEEDS_CHANNEL_COUNT__MASK) > NEEDS_CHANNEL_2,
  17.                         "Track %d needs downmix + resample", i);
  18.             } else {
  19.                 ......
  20.             }
  21.         }
  22.     }
  23.     // select the processing hooks
  24.     state->hook = process__nop;
  25.     if (countActiveTracks > 0) {
  26.         if (resampling) {
  27.             if (!state->outputTemp) {
  28.                 state->outputTemp = new int32_t[MAX_NUM_CHANNELS * state->frameCount];
  29.             }
  30.             if (!state->resampleTemp) {
  31.                 state->resampleTemp = new int32_t[MAX_NUM_CHANNELS * state->frameCount];
  32.             }
  33.             state->hook = process__genericResampling; /* 在需要重采样操作的情况下,调用process_genericResampling()函数*/
  34.         } else {
  35.             ......
  36.         }
  37.     }
  38.     ......
  39.     // Now that the volume ramp has been done, set optimal state and
  40.     // track hooks for subsequent mixer process
  41.     ......
  42. }
process_genericResampling()函数代码如下:
  1. // generic code with resampling
  2. void AudioMixer::process__genericResampling(state_t* state)
  3. {
  4.     ......
  5.     uint32_t e0 = state->enabledTracks;
  6.     while (e0) {
  7.         // process by group of tracks with same output buffer
  8.         // to optimize cache use
  9.         ......
  10.         while (e1) {
  11.             ......
  12.             // this is a little goofy, on the resampling case we don't
  13.             // acquire/release the buffers because it's done by
  14.             // the resampler.
  15.             if (t.needs & NEEDS_RESAMPLE) {
  16.                 t.hook(&t, outTemp, numFrames, state->resampleTemp, aux); /* 调用track__genericResample()函数执行Resample */
  17.             } else {
  18.                 ......
  19.             }
  20.         }
  21.         convertMixerFormat(out, t1.mMixerFormat,
  22.                 outTemp, t1.mMixerInFormat, numFrames * t1.mMixerChannelCount);
  23.     }
  24. }
至此,Android系统播放音频时的Resample过程就分析完成了。

具体的Resample处理实质是数字信号处理,是个数学运算过程。Android系统中提供的算法有线性插值三次插值FIR滤波 3种。感兴趣的工程师同仁可以自行查阅相关资料书籍,这里不对数字信号处理的细节进行讨论。

阅读(6761) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~