Android debugging using Lauterbach Trace32

Today I will briefly go over my thoughts on Android debugging using Lauterbach Trace32 software. This blog entry attempts to introduce a possible method of Trace32 JTAG debugging to developers working with the Linux kernel or Android framework and is generally not targeted towards application developers. Also, depending upon your specific use case and experience, you should check out various other opensource debugging tools like remote gdb. 

Most embedded systems developers are already familiar with Trace32’s use as an in-circuit emulator to debug embedded RTOS and applications. We at Qualcomm Innovation Center, Inc. (QuIC) have used Trace32 to debug modem software for a few years now. Linux Trace32 debugging is relatively newer. 

Folks already familiar with debugging modem software using Trace32 on Qualcomm chipsets need to be aware of some differences with Linux debugging, especially userspace debugging. Here are some key concepts to keep in mind when debugging Linux on Trace32: 

• Process Scheduling. Linux scheduler allocates CPU time slices to processes. The scheduler manages the time slices using timer interrupts and selects the next best task at each scheduling. This means that when stepping through the code in Trace32, you may see the program counter jumping to scheduler or another process. As a result, especially for userspace debugging, one has to be aware of which process you are debugging.  
• Virtual memory. 1G/3G virtual memory split is the most common memory split found in Linux systems. This means that top 1G virtual memory is reserved for Linux kernel and the remaining 3G is available for userspace processes. So while debugging, at any point the code from 0xC0000000 to 0xFFFFFFFF is kernel code, while 0×00 to 0xBFFFFFFF is userspace. The userspace code is process specific, so one cannot assume that you will hit the same code at a given address, as this will depend on the userspace process that is running.  
• Dynamic libraries. Shared libraries may be loaded dynamically at runtime. If you are debugging one of these libraries that is loaded at runtime, you will have to know the address at which the library loaded, and map the library symbols accordingly. Android prelinks commonly used libraries at predefined addresses in the virtual address space of the process. You can check out build/core/prelink-linux-arm.map in the Android build tree for this information. On target, you can also ‘cat /proc//maps’ to get information on libraries and their addresses that are loaded in a process’s context.  
• Loading symbols. Since the top 1G virtual memory is always used by Linux kernel, one can load Linux kernel symbols from vmlinux and the symbols in top 1G will always be valid. However when loading userspace symbols, one has to load them in process context. This can be specified as part of the data.load command.

A sample Trace32 script to set up Android debugging in Trace32 can be found in the target specific area at in the Android tree. The script takes root of the Android source tree as an argument and loads the Linux kernel symbols and some commonly used Android libraries’ symbols. This is a sample script, so feel free to customize it as per your needs. The script assumes you have a Lauterbach Trace32 attached to your target.

Here are some screenshots of Linux/Android Trace32 debugging session setup using debug_android script.

Figure 1 above shows a Trace32 session where JTAG is attached to a live target. Note that the program counter is at 0xC0169BC0, which is code in Linux kernel a per the virtual memory split described earlier.

Figure 2 above shows the Trace32 symbols window. Since Linux kernel symbols come from vmlinux and userspace library symbols come from different files, you can go up and down the symbol list to find the ones you are interested in inspecting.

Figure 3 above shows the Trace32 symbols window for libEGL.so.

Figure 4 above shows the high-level source code mapped to libEGL.so symbols. High-level code can be mapped using y.spath commands.

Figure 5 above shows an example of inspecting process global variables.

Figure 6 above shows a Trace32 session stopped at a breakpoint in the userspace code.

I hope this has been a useful introduction to get you started with Trace32 debugging with Linux/Android. Developers already familiar with Trace32 should be able to get started with the debug_android.cmm sample script and customize it per your debugging needs. I expect some folks may find Trace32 debugging quite useful, while others may find it overkill for their debugging scenarios.

Please do share your feedback on how Trace32 or other tools work out for your Linux/Android debugging needs.