Android 7.0 Audio的Resample过程详解

Qidi 2017.02.23 （Markdown & Haroopad）

---

【前言】

处理过音频文件的工程师都知道音频数据存在采样率（Sample Rate）这个指标。在位深度（Bit Depth）一定的情况下，采样率越高，理论上来说播放出来的声音就越细腻，录制的声音也就越保真，反之亦然。

但在较早的Android系统版本上，不管音频文件原来的采样率几何，统统都被重采样（Resample）到44.1KHz进行播放，录制的时候则是被固定为8KHz进行采样。尽管这样的处理方式被广大音质爱好者所诟病，但在当时它确实是一种实现设备兼容的有效方法。

作为Android Audio BSP工程师，有必要了解系统实现Resample的过程。现在Android系统已经发布到了7.0版本，一起看看在最新的版本上这个Resample的过程是怎样实现的吧。

【背景知识】

我们知道在Android系统中，当应用层APP播放一个音频文件时，Framework层的AudioPolicyService（APS）会接收上层APP传递来的音频参数（例如格式、声道、采样率等），并调用AudioFlinger的createTrack()方法对应创建1个Track，再调用openOutput()方法来打开1个outputStream，然后使用这个outputStream来创建相应的Playback线程（依据应用场景可以是OffloadThread、DirectOutputThread、MixerThread），最终在Playback线程中匹配之前创建的Track，开始自APP至HAL的数据传输。

【Resample过程分析】

那么我们对Android Audio Resample过程的分析就从AudioFlinger开始。在AudioFlinger::openOutput()中可以看到，在Playback线程被成功创建之后，即被加入到mPlaybackThreads向量中进行管理了。具体代码如下：

sp<AudioFlinger::PlaybackThread> AudioFlinger::openOutput_l(audio_module_handle_t module,
                                                            audio_io_handle_t *output,
                                                            audio_config_t *config,
                                                            audio_devices_t devices,
                                                            const String8& address,
                                                            audio_output_flags_t flags)
{
    ......

    AudioStreamOut *outputStream = NULL;
    status_t status = outHwDev->openOutputStream( // 打开1个outputStream
            &outputStream,
            *output,
            devices,
            flags,
            config,
            address.string());

    mHardwareStatus = AUDIO_HW_IDLE;

    if (status == NO_ERROR) {

        PlaybackThread *thread;
        if (flags & AUDIO_OUTPUT_FLAG_COMPRESS_OFFLOAD) {
            thread = new OffloadThread(this, outputStream, *output, devices, mSystemReady);
            ALOGV("openOutput_l() created offload output: ID %d thread %p", *output, thread);
        } else if ((flags & AUDIO_OUTPUT_FLAG_DIRECT)
                || !isValidPcmSinkFormat(config->format)
                || !isValidPcmSinkChannelMask(config->channel_mask)) {
            thread = new DirectOutputThread(this, outputStream, *output, devices, mSystemReady);
            ALOGV("openOutput_l() created direct output: ID %d thread %p", *output, thread);
        } else {
            thread = new MixerThread(this, outputStream, *output, devices, mSystemReady); /* 默认情况下，创建MixerThread类型的Playback线程 */
            ALOGV("openOutput_l() created mixer output: ID %d thread %p", *output, thread);
        }
        mPlaybackThreads.add(*output, thread); /* 将新创建的线程加入向量 */
        return thread;
    }

    return 0;
}

随后Playback线程运行，对应的AudioFlinger::Playback::threadLoop()方法被执行，在该方法中调用了prepareTracks_l()函数。这个函数实际上是对应于AudioFlinger::MixerThread::prepareTracks_l()这个函数。threadLoop()函数代码细节如下：

bool AudioFlinger::PlaybackThread::threadLoop()
{
    Vector< sp<Track> > tracksToRemove;
    ......

    while (!exitPending())
    {
        cpuStats.sample(myName);
        Vector< sp<EffectChain> > effectChains;
        { // scope for mLock
            Mutex::Autolock _l(mLock);
            processConfigEvents_l();
            ......
            saveOutputTracks();
            ......
            if ((!mActiveTracks.size() && systemTime() > mStandbyTimeNs) ||
                                   isSuspended()) {
                // put audio hardware into standby after short delay
                if (shouldStandby_l()) {

                    threadLoop_standby();

                    mStandby = true;
                }
                ......
            }
            // mMixerStatusIgnoringFastTracks is also updated internally
            mMixerStatus = prepareTracks_l(&tracksToRemove); /* 调用prepareTracks_l()，为Playback线程匹配已注册的Track */
            ......
            // prevent any changes in effect chain list and in each effect chain
            // during mixing and effect process as the audio buffers could be deleted
            // or modified if an effect is created or deleted
            lockEffectChains_l(effectChains);
        } // mLock scope ends
        ......
        // enable changes in effect chain
        unlockEffectChains(effectChains);
        // Finally let go of removed track(s), without the lock held
        // since we can't guarantee the destructors won't acquire that
        // same lock. This will also mutate and push a new fast mixer state.
        threadLoop_removeTracks(tracksToRemove);
        tracksToRemove.clear();
        // FIXME I don't understand the need for this here;
        // it was in the original code but maybe the
        // assignment in saveOutputTracks() makes this unnecessary?
        clearOutputTracks();
        // Effect chains will be actually deleted here if they were removed from
        // mEffectChains list during mixing or effects processing
        effectChains.clear();
        // FIXME Note that the above .clear() is no longer necessary since effectChains
        // is now local to this block, but will keep it for now (at least until merge done).
    }
    threadLoop_exit();
    if (!mStandby) {
        threadLoop_standby();
        mStandby = true;
    }
    ......
    return false;
}

Resample的过程就发生在prepareTracks_l()函数中，所以我们来好好阅读一下。在该函数中，通过一个for循环遍历所有处于active状态的track。每一次循环中，都要进行如下2步操作： 

1. 通过reqSampleRate = track->mAudioTrackServerProxy->getSampleRate()来获取硬件设备所支持的采样率； 
2. 之后调用mAudioMixer->setParameter(name, AudioMixer::RESAMPLE, AudioMixer::SAMPLE_RATE, (void*)(uintptr_t)reqSampleRate)，通过对比音频文件采样率和音频设备支持的采样率，判断是否创建新的Resampler对象，或者从已有的Resampler对象列表中返回1个；

prepareTracks_l()函数代码细节如下：

AudioFlinger::PlaybackThread::mixer_state AudioFlinger::MixerThread::prepareTracks_l(
        Vector< sp<Track> > *tracksToRemove)
{
    ......
    // find out which tracks need to be processed
    size_t count = mActiveTracks.size(); /* 获取处于active状态的track的数量 */
    ......
    for (size_t i=0 ; i<count ; i++) {
        const sp<Track> t = mActiveTracks[i].promote();
        if (t == 0) {
            continue;
        }
        // this const just means the local variable doesn't change
        Track* const track = t.get(); /* 获取对应的track */
        ......
        audio_track_cblk_t* cblk = track->cblk();

        // The first time a track is added we wait
        // for all its buffers to be filled before processing it
        int name = track->name();
        ......
        if ((framesReady >= minFrames) && track->isReady() &&
                !track->isPaused() && !track->isTerminated())
        {
            ......
            int param = AudioMixer::VOLUME;
            if (track->mFillingUpStatus == Track::FS_FILLED) {
                // no ramp for the first volume setting
                track->mFillingUpStatus = Track::FS_ACTIVE;
                if (track->mState == TrackBase::RESUMING) {
                    track->mState = TrackBase::ACTIVE;
                    param = AudioMixer::RAMP_VOLUME;
                }
                mAudioMixer->setParameter(name, AudioMixer::RESAMPLE, AudioMixer::RESET, NULL);
            // FIXME should not make a decision based on mServer
            } else if (cblk->mServer != 0) {
                // If the track is stopped before the first frame was mixed,
                // do not apply ramp
                param = AudioMixer::RAMP_VOLUME;
            }

            // compute volume for this track
            ......
            // Delegate volume control to effect in track effect chain if needed
            ......

            // XXX: these things DON'T need to be done each time
            mAudioMixer->setBufferProvider(name, track);
            mAudioMixer->enable(name);

            mAudioMixer->setParameter(name, param, AudioMixer::VOLUME0, &vlf); /* 设置左声道音量 */
            mAudioMixer->setParameter(name, param, AudioMixer::VOLUME1, &vrf); /* 设置右声道音量 */
            mAudioMixer->setParameter(name, param, AudioMixer::AUXLEVEL, &vaf); /* 设置辅助声道音量 */
            mAudioMixer->setParameter(
                name,
                AudioMixer::TRACK,
                AudioMixer::FORMAT, (void *)track->format()); /* 设置音频数据格式 */
            mAudioMixer->setParameter(
                name,
                AudioMixer::TRACK,
                AudioMixer::CHANNEL_MASK, (void *)(uintptr_t)track->channelMask()); /* 设置音频声道数 */
            mAudioMixer->setParameter(
                name,
                AudioMixer::TRACK,
                AudioMixer::MIXER_CHANNEL_MASK, (void *)(uintptr_t)mChannelMask);
            // limit track sample rate to 2 x output sample rate, which changes at re-configuration
            uint32_t maxSampleRate = mSampleRate * AUDIO_RESAMPLER_DOWN_RATIO_MAX;
            uint32_t reqSampleRate = track->mAudioTrackServerProxy->getSampleRate(); /* 获取音频设备所支持的采样率 */
            if (reqSampleRate == 0) {
                reqSampleRate = mSampleRate;
            } else if (reqSampleRate > maxSampleRate) {
                reqSampleRate = maxSampleRate;
            }
            mAudioMixer->setParameter(
                name,
                AudioMixer::RESAMPLE,
                AudioMixer::SAMPLE_RATE, /* 设置音频采样率(必要时会进行重采样) */
                (void *)(uintptr_t)reqSampleRate);

            AudioPlaybackRate playbackRate = track->mAudioTrackServerProxy->getPlaybackRate();
            mAudioMixer->setParameter(
                name,
                AudioMixer::TIMESTRETCH,
                AudioMixer::PLAYBACK_RATE, /* 设置播放码率 */
                &playbackRate);

            /*
             * Select the appropriate output buffer for the track.
             *
             * Tracks with effects go into their own effects chain buffer
             * and from there into either mEffectBuffer or mSinkBuffer.
             *
             * Other tracks can use mMixerBuffer for higher precision
             * channel accumulation. If this buffer is enabled
             * (mMixerBufferEnabled true), then selected tracks will accumulate
             * into it.
             *
             */
            if (mMixerBufferEnabled
                    && (track->mainBuffer() == mSinkBuffer
                            || track->mainBuffer() == mMixerBuffer)) {
                mAudioMixer->setParameter(
                        name,
                        AudioMixer::TRACK,
                        AudioMixer::MIXER_FORMAT, (void *)mMixerBufferFormat); /* 设置缓冲区数据格式 */
                mAudioMixer->setParameter(
                        name,
                        AudioMixer::TRACK,
                        AudioMixer::MAIN_BUFFER, (void *)mMixerBuffer); /* 分配主缓冲区 */
                // TODO: override track->mainBuffer()?
                mMixerBufferValid = true;
            } else {
                ......
            }
            mAudioMixer->setParameter(
                name,
                AudioMixer::TRACK,
                AudioMixer::AUX_BUFFER, (void *)track->auxBuffer()); /* 分配副缓冲区 */

            // reset retry count
            track->mRetryCount = kMaxTrackRetries;

            // If one track is ready, set the mixer ready if:
            // - the mixer was not ready during previous round OR
            // - no other track is not ready
            if (mMixerStatusIgnoringFastTracks != MIXER_TRACKS_READY ||
                    mixerStatus != MIXER_TRACKS_ENABLED) {
                mixerStatus = MIXER_TRACKS_READY;
            }
        } else {
            // 出现underrun，以及相应处理操作
            ......
        }
    }

    // Push the new FastMixer state if necessary
    ......
    // Now perform the deferred reset on fast tracks that have stopped
    ......
    // remove all the tracks that need to be...
    removeTracks_l(*tracksToRemove);
    ......
    // sink or mix buffer must be cleared if all tracks are connected to an
    // effect chain as in this case the mixer will not write to the sink or mix buffer
    // and track effects will accumulate into it
    ......
    // if any fast tracks, then status is ready
    ......
    return mixerStatus;
}

在确认要使用的Resampler对象存在后，调用invalidateState(1 << name)使设置生效，开始执行重采样。invalidateState()函数会调用AudioMixer::process_validate()，在该函数中首先通过语句t.hook = getTrackHook(TRACKTYPE_RESAMPLE, t.mMixerChannelCount, t.mMixerInFormat, t.mMixerFormat);获取执行重采样操作的函数，随后通过state->hook = process_resampling;中的t.hook(&t, outTemp, numFrames, state->resampleTemp, aux)语句进行调用。 

setParameter()函数代码如下：

void AudioMixer::setParameter(int name, int target, int param, void *value)
{
    ......
    int valueInt = static_cast<int>(reinterpret_cast<uintptr_t>(value));
    int32_t *valueBuf = reinterpret_cast<int32_t*>(value);
    switch (target) {
    ......
    case RESAMPLE:
        switch (param) {
        case SAMPLE_RATE:
            ALOG_ASSERT(valueInt > 0, "bad sample rate %d", valueInt);
            if (track.setResampler(uint32_t(valueInt), mSampleRate)) { /* 新建或查找1个Resampler对象 */
                ALOGV("setParameter(RESAMPLE, SAMPLE_RATE, %u)",
                        uint32_t(valueInt));
                invalidateState(1 << name); /* 使设置生效，调用重采样的后续处理函数 */
            }
            break;
        case RESET:
            track.resetResampler();
            invalidateState(1 << name);
            break;
        case REMOVE:
            delete track.resampler;
            track.resampler = NULL;
            track.sampleRate = mSampleRate;
            invalidateState(1 << name);
            break;
        default:
            LOG_ALWAYS_FATAL("setParameter resample: bad param %d", param);
        }
        break;
    }
}

invalidateState()函数代码如下：

void AudioMixer::invalidateState(uint32_t mask)
{
    if (mask != 0) {
        mState.needsChanged |= mask;
        mState.hook = process__validate;
    }
}

process__validate()函数代码如下：

void AudioMixer::process__validate(state_t* state)
{
    ......
    uint32_t en = state->enabledTracks;
    while (en) {
        ......
        if (n & NEEDS_MUTE) {
            ......
        } else {
            ......
            if (n & NEEDS_RESAMPLE) {
                all16BitsStereoNoResample = false;
                resampling = true;
                t.hook = getTrackHook(TRACKTYPE_RESAMPLE, t.mMixerChannelCount,
                        t.mMixerInFormat, t.mMixerFormat); /* 获取track__genericResample()函数 */
                ALOGV_IF((n & NEEDS_CHANNEL_COUNT__MASK) > NEEDS_CHANNEL_2,
                        "Track %d needs downmix + resample", i);
            } else {
                ......
            }
        }
    }
    // select the processing hooks
    state->hook = process__nop;
    if (countActiveTracks > 0) {
        if (resampling) {
            if (!state->outputTemp) {
                state->outputTemp = new int32_t[MAX_NUM_CHANNELS * state->frameCount];
            }
            if (!state->resampleTemp) {
                state->resampleTemp = new int32_t[MAX_NUM_CHANNELS * state->frameCount];
            }
            state->hook = process__genericResampling; /* 在需要重采样操作的情况下，调用process_genericResampling()函数*/
        } else {
            ......
        }
    }
    ......
    // Now that the volume ramp has been done, set optimal state and
    // track hooks for subsequent mixer process
    ......
}

process_genericResampling()函数代码如下：

// generic code with resampling
void AudioMixer::process__genericResampling(state_t* state)
{
    ......
    uint32_t e0 = state->enabledTracks;
    while (e0) {
        // process by group of tracks with same output buffer
        // to optimize cache use
        ......
        while (e1) {
            ......
            // this is a little goofy, on the resampling case we don't
            // acquire/release the buffers because it's done by
            // the resampler.
            if (t.needs & NEEDS_RESAMPLE) {
                t.hook(&t, outTemp, numFrames, state->resampleTemp, aux); /* 调用track__genericResample()函数执行Resample */
            } else {
                ......
            }
        }
        convertMixerFormat(out, t1.mMixerFormat,
                outTemp, t1.mMixerInFormat, numFrames * t1.mMixerChannelCount);
    }
}

至此，Android系统播放音频时的Resample过程就分析完成了。

具体的Resample处理实质是数字信号处理，是个数学运算过程。Android系统中提供的算法有线性插值、三次插值、FIR滤波 3种。感兴趣的工程师同仁可以自行查阅相关资料书籍，这里不对数字信号处理的细节进行讨论。