Debug a Crash

Introduction

You may configure your CAT-DEV to generate a full core dump, post-process the core dump into an ELF file, and then view your application's state at the time of the crash in MULTI. See Debugging a Crash Dump below for details. MULTI version 5.3.22 or later is required to view the resulting ELF file.

You may also configure your CAT-DEV to capture the contents of the /usr/tmp/app directory when a crash occurs. See Capturing /usr/tmp/app below for details.

Debugging a Crash Dump

The viewcore command creates an ELF file from a COS core dump and then launches the MULTI debugger to view the crash state.

To debug a crash dump with viewcore using PCFS

  1. Run cafeon to start the System Config Tool.
  2. In the System Config Tool, select Other Setting.
  3. In Crash Dump Type, select Full.
  4. When a crash occurs, the following messages are printed:
  5. 00:00:08:477: CBL Trace: Saving COS block log files to PCFS:/save/osblocklog/jan29_024213.
    00:00:08:477: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/OSEventLog.bin.
    00:00:08:536: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/OSEventLog.bin [8%].
    ...
    00:00:09:188: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/OSEventLog.bin [100%].
    00:00:09:195: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK3_A50000000_SZ71fa0000.bin.
    00:00:20:756: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK3_A50000000_SZ71fa0000.bin [5%].
    ...
    00:03:44:110: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK3_A50000000_SZ71fa0000.bin [100%].
    00:03:44:117: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK4_Ac2000000_SZ0e000000.bin.
    00:03:45:438: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK4_Ac2000000_SZ0e000000.bin [5%].
    ...
    00:04:10:409: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK4_Ac2000000_SZ0e000000.bin [100%].
    00:04:10:416: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK7_A1bf80000_SZ00060000.bin.
    00:04:10:446: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK7_A1bf80000_SZ00060000.bin [66%].
    00:04:10:466: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK7_A1bf80000_SZ00060000.bin [100%].
    00:04:10:473: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK8_A1bb01000_SZ00001000.bin.
    00:04:10:481: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK8_A1bb01000_SZ00001000.bin [100%].
    00:04:10:488: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK9_A32000000_SZ007e0000.bin.
    00:04:10:547: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK9_A32000000_SZ007e0000.bin [6%].
    ...
    00:04:11:421: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK9_A32000000_SZ007e0000.bin [100%].
    00:04:11:428: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK10_A18000000_SZ03000000.bin.
    00:04:11:724: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK10_A18000000_SZ03000000.bin [5%].
    ...
    00:04:17:071: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK10_A18000000_SZ03000000.bin [100%].
    00:04:17:078: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK11_A08004198_SZ000000b8.bin.
    00:04:17:085: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK11_A08004198_SZ000000b8.bin [100%].
    00:04:17:091: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK12_A08044198_SZ000000b8.bin.
    00:04:17:100: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK12_A08044198_SZ000000b8.bin [100%].
    00:04:17:107: CBL Trace: Creating OS memory block save file PCFS:/save/osblocklog/jan29_024213/BLK13_A08084198_SZ000000b8.bin.
    00:04:17:113: CBL Trace: Saved PCFS:/save/osblocklog/jan29_024213/BLK13_A08084198_SZ000000b8.bin [100%].
    00:04:17:123: CBL Trace: In total, 10 OS memory blocks were saved.
    00:04:17:123: CBL Trace: IOS_CRASH_START_PROCESS:viewcore -x ${CAFE_SAVE_DIR}/osblocklog/jan29_024213
    
  6. Your PC should have a $CAFE_SAVE_DIR/osblocklog directory. Inside osblocklog, a directory named mmmdd_hhmmss is created to hold the various files, where mmm, dd, hh, mm, and ss are the month, day, hour, minute and second of the crash. There should be 9 memory block files and an OSEventLog.bin file.
  7. The system should automatically invoke viewcore to generate the ELF file core and launch the MULTI debugger. Since block 3 is a large file, it takes longer time to write its segment to the ELF file than the other blocks. The following messages are printed as the ELF file is written:
    Creating Program Segment for C:/dev/cafe/data/save/osblocklog/dec26_233812/BLK11_A08004198_SZ000000b8.bin...
      Done
    ...
    Creating Program Segment for C:/dev/cafe/data/save/osblocklog/dec26_233812/BLK3_A50000000_SZ71fa0000.bin...
      Writing 83886080 bytes,  4%
      ...
      Writing 1845493760 bytes, 97%
      Done
    MULTI v5.3.22   is launching core
    
  8. You may now examine the state at the time of the crash.

Managing the Number of Crash Dumps

You may control the number of crash dumps that are maintained on the PC. If you set the PC environment variable CAFE_CRASH_DUMP_LIMIT to a number greater than zero, say n, then viewcore retains the most recent n crash dumps. If you set CAFE_CRASH_DUMP_LIMIT to 0, then all crash dumps will be retained. By default, n = 5.

NOTE:
viewcore management rules will delete the oldest directory at $CAFE_SDK_ROOT/data/save/osblocklog. viewcore regards any directory in this location as a crash dump. To save a crash dump or directory that exists at this location, beyond the viewcore management rules, copy the crash dump or directory to a location other than $CAFE_SDK_ROOT/data/save/osblocklog.

Disabling the Launch of MULTI

By default, after a crash dump is uploaded via PCFS and converted using viewcore, MULTI starts so that you may debug the crash.

To prevent MULTI from starting after the upload, set the CAFE_CRASH_DUMP_NOMULTI environment variable to 1. To allow MULTI to start again, set CAFE_CRASH_DUMP_NOMULTI to 0 or unset it.

To debug a crash dump with viewcore using an SD card

Use these steps if PCFS is not available.

NOTE:
For best dump performance, use a Class 10 SD card.
  1. Format an SD card on your host PC using FAT32. The SD card must have a minimum of 3 GB free per COS core dump.
  2. Insert the SD card into the CAT-DEV's SD card slot.
  3. Run cafeon to start the System Config Tool.
  4. In the System Config Tool, select Other Setting.
  5. In Crash Dump Type, select Full.
  6. When a crash occurs and the SD card is in place, the following messages are printed:
  7. CBL Trace: Saving application files to SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918.
    00:00:39:539: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/OSEventLog.bin.
    00:00:39:587: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/OSEventLog.bin [8%].
    ...
    00:00:40:219: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/OSEventLog.bin [100%].
    00:00:40:465: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK3_A50000000_SZ71fa0000.bin.
    00:00:47:450: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK3_A50000000_SZ71fa0000.bin [5%].
    ...
    00:03:00:373: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK3_A50000000_SZ71fa0000.bin [100%].
    00:03:15:571: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK4_Ac2000000_SZ0e000000.bin.
    00:03:16:439: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK4_Ac2000000_SZ0e000000.bin [5%].
    ...
    00:03:32:766: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK4_Ac2000000_SZ0e000000.bin [100%].
    00:03:34:937: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK7_A1bf80000_SZ00060000.bin.
    00:03:34:957: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK7_A1bf80000_SZ00060000.bin [66%].
    00:03:34:972: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK7_A1bf80000_SZ00060000.bin [100%].
    00:03:35:417: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK8_A1bb01000_SZ00001000.bin.
    00:03:35:427: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK8_A1bb01000_SZ00001000.bin [100%].
    00:03:35:862: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK9_A32000000_SZ007e0000.bin.
    00:03:35:910: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK9_A32000000_SZ007e0000.bin [6%].
    ...
    00:03:36:564: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK9_A32000000_SZ007e0000.bin [100%].
    00:03:37:040: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK10_A18000000_SZ03000000.bin.
    00:03:37:268: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK10_A18000000_SZ03000000.bin [5%].
    ...
    00:03:41:224: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK10_A18000000_SZ03000000.bin [100%].
    00:03:41:885: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK11_A08004198_SZ000000b8.bin.
    00:03:41:892: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK11_A08004198_SZ000000b8.bin [100%].
    00:03:42:329: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK12_A08044198_SZ000000b8.bin.
    00:03:42:338: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK12_A08044198_SZ000000b8.bin [100%].
    00:03:42:771: CBL Trace: Creating OS memory block save file SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK13_A08084198_SZ000000b8.bin.
    00:03:42:778: CBL Trace: Saved SD:/private/Nintendo Wii U/sys/osblocklog/jan29_021918/BLK13_A08084198_SZ000000b8.bin [100%].
    00:03:43:422: CBL Trace: In total, 10 OS memory blocks were saved.
    
  8. Remove the SD card from the CAT-DEV and connect it to your host PC.
  9. The SD card should have an osblocklog directory. Inside osblocklog, a directory named mmmdd_hhmmss is created to hold the various files, where mmm, dd, hh, mm, and ss are the month, day, hour, minute and second of the crash. There should be 4 memory block files and an OSEventLog.bin file.
  10. Change your directory to the mmmdd_hhmmss directory with the desired COS core dump files.
  11. Invoke viewcore to generate the ELF file core and launch the MULTI debugger. Since block 3 is a large file, it takes longer time to write its segment to the ELF file than the other blocks.
  12. viewcore.
    
    The following messages are printed as the ELF file is written:
    Creating Program Segment for BLK8_A1bb01000_SZ00001000.bin...
    Done
    Creating Program Segment for BLK7_A1bf80000_SZ00060000.bin...
    Done
    Creating Program Segment for BLK4_Ac2000000_SZ0e000000.bin...
    Done
    Creating Program Segment for BLK3_A50000000_SZ71fa0000.bin...
    Writing 83886080 bytes,  4%
    ...
    Writing 1845493760 bytes, 97%
    Done
    
  13. You may now examine the state at the time of the crash.

Capturing /usr/tmp/app

To capture the contents of /usr/tmp/app upon a crash using PCFS

  1. Configure your CAT-DEV to boot in PCFS mode. See Change CAT-DEV Boot Modes for details.
  2. Run cafeon to start the System Config Tool.
  3. In Other Setting, select Copy Tmp on Crash.
  4. In Copy Tmp on Crash, select Enable.
  5. When a crash occurs, you should see console messages similar to the following.
    00:03:35:209: AFS Trace: Saving application files to PCFS:/save/osfilelog/jan29_184218.
    00:03:35:212: AFS Trace: Looking for files in /vol/storage_mlc01/usr/tmp/app.
    00:04:08:115: AFS Trace: Found 111 subdirectories in /vol/storage_mlc01/usr/tmp/app. Copied all 283 files.
    
    
  6. Your host PC should have a $CAFE_SAVE_DIR/osfilelog directory. Inside should be a directory named mmmdd_hhmmss, where mmm, dd, hh, mm, and ss are the month, day, hour, minute and second of the crash, respectively. This directory should contain the files from /usr/tmp/app at the time of the crash.

To capture the contents of /usr/tmp/app upon a crash using an SD card

Use these steps if PCFS is not available.

  1. Format an SD card on your host PC using FAT32.
  2. Insert the SD card into your CAT-DEV's SD card slot.
  3. Run cafeon to start the System Config Tool.
  4. In Other Setting, select Copy Tmp on Crash.
  5. In Copy Tmp on Crash, select Enable.
  6. When a crash occurs, you should see console messages similar to the following.
    00:07:02:566: AFS Trace: Saving application files to SD:/private/Nintendo Wii U/sys/osfilelog/jan29_013329.
    00:07:02:725: AFS Trace: Looking for files in /vol/storage_mlc01/usr/tmp/app.
    00:08:05:386: AFS Trace: Found 111 subdirectories in /vol/storage_mlc01/usr/tmp/app. Copied all 283 files.
    00:08:06:164: AFS Trace: Saving application files to SD:/private/Nintendo Wii U/sys/osfilelog/jan29_013433.
    
  7. Remove the SD card from the CAT-DEV and connect it to your host PC.
  8. The SD card should have an /private/Nintendo Wii U/sys/osfilelog directory. Inside should be a directory named mmmdd_hhmmss, where mmm, dd, hh, mm, and ss are the month, day, hour, minute and second of the crash, respectively. This directory should contain the files from /usr/tmp/app at the time of the crash.

See Also

pcel: Parse OS Event Log
viewcore: View Core Dump

Revision History

2014/03/12 Revisions to CBL and AFS paths.
2014/03/04 Added viewcore management note.
2014/02/18 Add viewcore environment variables.
2014/02/04 Add instructions for capturing /usr/tmp.
2013/11/13 Recommend Class 10 SD Card for best performance.
2013/09/15 Initial version.


CONFIDENTIAL