Android Recent Tasks XML Parser

This post is a continuation of my last blog post where I introduced a simple parser for the Android usagestats XML files.

https://abrignoni.blogspot.com/2019/02/android-usagestats-xml-parser.html

In this entry I am introducing a parser for the Android recent tasks XML files. Like the previous parser it is based on the research done by Jessica Hyde that she presented at the SANS DFIR Summit 2018. You can see her excellent presentation here:

YouTube: Every Step You Take: Application and Network Usage in Android 

The presentation slides, in PDF format, can be found here:

PDF Slides: Every Step You Take: Application and Network Usage in Android 

As explained in the presentation the Recent Tasks XML files record the following activities for recently used apps:

• Task ID number = Used to correlate snapshot and recent image files.
• Effective UID = App identifier.
• First active time = Timestamp in millisecond epoch time.
• Last active time= Timestamp in millisecond epoch time.
• Last time moved = Timestamp in millisecond epoch time. 
• Affinity = Bundle ID name.
• Calling package = Bundle ID or process that called the referenced recent task.
• Real activity = Gives information on app usage at time of recording and snapshot creation.

These XML files are located in the following directory:

\system_ce\0\recent_tasks

In addition to these XML files, recent tasks can produce snapshot images as well as recent images. Details about these are contained in the previously referenced presentation. These images can be found in the corresponding directories:

\system_ce\0\shortcut_service\snapshots

\system_ce\0\recent_images

In order to leverage the data contained in these XML files and images I made a parser in Python 3 that takes the XML information and puts it in a SQLite database for ease of querying. The script can be found here: 

https://github.com/abrignoni/Android-Recent-Tasks-XML-Parser

Warning:

The script has been tested and found to be accurate on my own data sets. Not all recent tasks will contain all data events or related images. Additional testing and validation of the script is humbly requested and more than welcomed.

Script usage

1. Extract from your Android source device the three directories mentioned previously. Extraction should be logical and not contained forensic tool generated recovered items like deleted and/or file slack files.

2. Place the script and the noimage.jpg files from the repository in the same root directory as the extracted directories.

Have this before running script.

3. Run the script with no arguments.

Script is done.

4. When completed the script will generate two files, a SQLite database named RecentAct.db and a report file named Recent_Activity.html.

What you should see after a successful run of the script.

Note that the RecentAct.db SQLite file will contain two fields populated with all the XML attributes in JSON format. The analyst can run a query using JSON_extract to custom generate queries with any of the attributes within the XML.

5. Open the Recent_Activity.html report.

Sample report entry.

For every recent task there will be a table with pertinent information as well as the snapshot and recent image files that correspond to it. To view the images full size just click on them. Be aware of the importance of the creation times of these image files within the source media. For details see the presentation previously mentioned.

It is of note that not all recent tasks, in some of my test data samples, had corresponding images or full sets of attributes. When a recent task lacks corresponding images the script will reference the noimg.jp file.

Missing image and missing attributes.

For missing attributes the report will state 'NO DATA' and/or 'NO IMAGE' in the Key and Values columns as needed. Be aware that the SQLite database has all attributes in JSON format for custom query generation.

Conclusion

I want to thank again Jessica Hyde for her research and for making the community aware of these artifacts. Hopefully this script can make it easier to give much needed context to these images and apps whose value might not be found anywhere else on the source device.

As always I can be reached on twitter @AlexisBrignoni and email 4n6[at]abrignoni[dot]com.

Android Usagestats XML Parser

As I've been testing and using Sarah Edwards' excellent APOLLO pattern of life framework for iOS I reminded myself of the great work done by Jessica Hyde on a similar set of files for Android called usagestats. These files provide insight on what apps where being used, if they were in the foreground or background, and how long have the apps been active among many other forensically interesting data points.

In her presentation for the SANS DFIR Summit 2018, Jessica Hyde explains how the usagestats XML files record the following activity from Android devices:

• User interaction
• Move to foreground
• Move to background
• Configuration changes

I highly recommend the reader check out her Every Step you Take DFIR Summit video and presentation slides on PDF format. The rest of the blog post will make more sense after the viewing and/or reading of her work.

In order to leverage the data contained in these XML files I made a parser in Python 3 that takes the XML information and puts it in a SQLite database for ease of querying. The script can be found here: 

https://github.com/abrignoni/Android-Usagestats-XML-Parser

Warning:
The script has been tested and found to be accurate on my own data sets. Additional testing and validation of the script is humbly requested and more than welcomed.

Script usage

Extract from your source Android device the following directory:

\data\system\usagestats\

Export the usagestats directory

 Place the script at the in the same root directory as the just extracted usagestats directory.

Side by side as such.

Run the script with no arguments from the root directory that contains both the script and the usagestats directory. The script will parse all the internal directories and files for you.

No arguments are needed.

The script will also alert you of files whose content is not XML that can be parsed. After the script ends a sqlite database named usagestats.db is generated.

Data is served.

Use your favorite SQLite application to view the parsed data. Notice the timestamps are in epoch time.

Notice the fields.

The generated database contains a table named data with the following fields:

Usage_type: 

Each XML file contains a description of what type of data is recording. The values can be event-log, configuration, and packages.

Lastime:

Records when an app (package) was last active or when a configuration took place. The XMLs themselves keep track of these time in two ways. Most usage events maintain a count of how many milliseconds have passed since the creation of the XML file and the occurrence of the event. To calculate the timestamp of the event the script takes the XML filename, which is the epoch time of the file itself in milliseconds, and adds to it the milliseconds the event took to occur. This provides the event time as en epoch timestamp. For events that are not millisecond offsets from the epoch time filename of the XML file, they keep the time as en epoch timestamp preceded by a minus sign. The script eliminates the minus sign to keep the epoch timestamp. My testing has shown this way of calculating times to be accurate to activity I have taken on the device.

The following image was included in an app review I did in November for the TikTok Android application. Notice the time some of the chat activity took place.

Notice the created time

As seen in the image above the TikTok app is sending and receiving chat messages. The parsed XML SQLite database shows the same activity at the same time being totally consistent with TikTok chatting.

Notice the classs values

Shortly I will provide a SQL query that will format the dates and type values in the same human readable format seen above.

Time_active
Certain events keep track of their length in milliseconds.

Package:
Application name.

Types:
Activity types as integer values. These represent activity like move to background or move to foreground. The list of interactions can be found here:

https://developer.android.com/reference/android/app/usage/UsageEvents.Event

Classs:
Application name and corresponding modules in use.

Source:
Usagestat originating XML category. They are daily, weekly, monthly, and yearly.

Fullatt:
Contains the full attributes for the XML event, in other words all the data for the even in JSON format. With the data in this field the analyst can easily select any key:value pair and make it its own column in a SQL query by the use of JSON_Extract. For an example on how this SQL query function works see here:

https://abrignoni.blogspot.com/2018/09/finding-slack-messages-in-android-and.html

SQL query

The following query can be run against the script generated database to format the timestamps from UTC to local time, add a field for time_active in seconds, and changes the types integer values to readable activity descriptions. Be aware that I have not added all case types per the link provided previously in the Types section. Add as needed.

SELECT
  usage_type,
  datetime(lastime/1000, 'UNIXEPOCH', 'localtime') as lasttimeactive,
  timeactive as time_Active_in_msecs,
  timeactive/1000 as timeactive_in_secs,
  package,
CASE types
     WHEN '1' THEN 'MOVE_TO_FOREGROUND'
     WHEN '2' THEN 'MOVE_TO_BACKGROUND'
     WHEN '5' THEN 'CONFIGURATION_CHANGE'
     WHEN '7' THEN 'USER_INTERACTION'
     WHEN '8' THEN 'SHORTCUT_INVOCATION'
     ELSE types
END types,
  classs,
  source,
  fullatt
FROM data
ORDER BY lasttimeactive DESC

Conclusion

My hope with this script is to make accessible the data contained in the usagestats xml files for digital forensic case work. Additional testing of the script and suggestions on how to optimize it are welcomed. I hope to create additional scripts that will parse the Android battery status and recent tasks XML files as show by Jessica Hyde in her presentation.

As always I can be reached on twitter @AlexisBrignoni and email 4n6[at]abrignoni[dot]com.