AWS EC2 Cloud Storage Acquisition with Evimetry

You have been tasked with forensic acquisition of 6 servers in the AWS cloud, with a total of 2TB of storage. How do you do it?

This post will describe the method I applied in a recent case, where we collect the storage, acquired it into forensic images, and pulled down the images into our custody overnight. While I will be describing how I did it using Evimetry, the method is easily translatable to other tools.

Storage forensics in AWS

Unlike many cloud IAAS platforms, AWS provides us with the ability to take a Snapshot of the storage of a Virtual Computer (an Instance, in EC2 parlance). This gives you a point in time copy of the storage device. This isn’t a forensic image, as there isn’t a hash protecting the copy.

This enables us to quickly Collect a copy of the storage, without affecting the availability of the Target device. To truly collect the copy though, we need to take it under our control.  To do this, we rely on the ability of AWS to share Snapshots between accounts.

Once we have access to the Snapshot in our own Security Domain (account), we can then shift to forensic acquisition of the copy. This is best achieved by generating a Volume from the Snapshot, attaching the Volume to a purpose-built acquisition server, and acquiring using regular forensic processes. Once a forensic image is acquired, we then Transfer it out of the cloud to store on a storage device that we Possess.

1-EC2-CloudStorage

The following sections step through the process of undertaking the method.

Evidence Isolation & Location

First up, create your own Security Domain for Collecting the Snapshots into, and undertaking acquisition. In AWS, this is easily achieved by maintaining your own account, separate to the TARGET account. The below screenshot displays an account I have established under my own name, logged into the AWS Console.

I recommend running the two separate AWS security domains (the TARGET and EVIDENCE) using two separate web browser windows, one of them using private mode browsing so that you can use the TARGET’s credentials in one, and your security domain’s credentials in another.

2-AccountSettings

Note the Account ID (993480464498) – this will be required to identify this security domain when we come to share a Snapshot from the TARGET to our security domain.

Identification

In the TARGET AWS console, use the left menu, “Instances” to show the instances, and find the instance that you want to collect.

In the screen capture below, one can see the TARGET instance, the instance ID (in this case i-065e4cd1fbf56c92e), the block device volume (vol-08c5f1566ec4ea6c5), and the Availability Zone “ap-southeast-2c”. These identifiers should be documented to establish the provenance of the evidence.

3-Target-Details

 

In the left menu, “Elastic Block Store”, select “Snapshots”, and then “Create Snapshot”.

4-CreateSnapshot

 

Select the volume of our TARGET server (ol-08c5f1566ec4ea6c5), and describe the evidence.

5-CreateSnapshot

We now have a snapshot of the block storage of the instance. We record the Snapshot ID (snap-0925cec0faee0659a) to maintain provenance of the evidence.

6-ModifyPermissions

Now that we have an image (not a forensic image, as we don’t have a hash), we want to Collect it. This means taking possession of it, so that it can’t be modified. To do this, we share the image with the EVIDENCE security domain we created earlier.

Recalling that the Account ID of our acquisition security domain is (993480464498), we privately share the image with that account.

7-ModifyPermissions

Note that the snapshot isn’t instantaneous and may take some minutes to complete.

Prepare Evidence Storage Server – Server provisioning

While the snapshot is going, switch browser windows (and security domains), and begin setting up your Evidence Server in the same datacentre as the target server. This will be running the Evimetry Cloud Agent, and will be co-located with the TARGET server for efficiency and speed. In the below AWS control panel, we set the location to “Sydney” which matches the TARGET server in this instance.

In the left menu we select “Instances” and then “Launch Instance”

8-LaunchInstance

To deploy the Evimetry Cloud Agent, we need an Ubuntu 14.04 instance.  Select that.

9-ChooseAMI

The speed at which your acquisition will occur will depend on a number of factors, including the virtual disk size, the performance of the virtual storage, and the number of CPU’s you have in the server. In a future blog post, I will go into this in more detail, but for now, select a 4 CPU machine with moderate performance.

10-ChooseInstanceType

Next up, we want to make sure that the evidence storage server is as close as possible to the target. From before, we have identified that that the target is in “ap-southeast-2c”, so we make sure that the subnet matches. We also ensure that we enable the auto assignment of a public IP, so we can connect to the server. This is sufficient to then “Review & Launch”.

11-ConfigureInstanceDetails

Finally, we launch the new Evidence Storage instance.

12-ReviewAndLaunch

The final task in bringing up the evidence storage server is to establish a key pair for working with the server. We create a new key pair called “SF-Acquisition” below.

13-CreateKeyPair

Download the key pair, and save it somewhere safely. Example shell commands follow.

cp ~/Downloads/SF-Acquisition.pem ~/Documents/keys/

chmod og-rwx ~/Documents/keys/SF-Acquisition.pem

After launching, traverse the launch status window to the new instance.

14-LaunchStatus

 

On following the instance link, we see the Instance details. Note the following important details. The public IP is 54.206.15.84, and the Security Group is “launch-wizard-3”.

15a-InstanceProperties

Prepare Evidence Server – Deploy Evimetry Cloud Agent

We now deploy the Evimetry Cloud Agent on the Evidence Server.

First up, we SSH into the Evidence Server, using the private key from before, and the public IP address of the machine.

neon:tmp bradley$ ssh -i ~/Documents/keys/SF-Acquisition.pem ubuntu@54.206.15.84

Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-125-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Fri Sep 15 07:01:38 UTC 2017

  System load:  0.0               Processes:           140

  Usage of /:   11.8% of 7.74GB   Users logged in:     0

  Memory usage: 0%                IP address for eth0: 172.31.19.255

  Swap usage:   0%

  Graph this data and manage this system at:

    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:

    http://www.ubuntu.com/business/services/cloud

19 packages can be updated.

8 updates are security updates.

New release ’16.04.3 LTS’ available.

Run ‘do-release-upgrade’ to upgrade to it.

The programs included with the Ubuntu system are free software;

the exact distribution terms for each program are described in the

individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by

applicable law.

ubuntu@ip-172-31-19-255:~$

We need to do admin level install operations, so make sure you are in a root shell session by executing “sudo bash”

ubuntu@ip-172-31-19-255:~$ sudo bash

root@ip-172-31-19-255:~#

Next, we install the Evimetry Cloud Agent, using a simple 2-line UNIX command. First log into the Evimetry Portal using your credentials. Then select the “Deploy Cloud Agent” option from the menu.

16-EvimetryPortal

We deploy the cloud agent using a simple 2 line shell script. Copy and paste it into your SSH session.

17-DeployCloudAgent

In the SSH session, the VM will be patched, and the Evimetry Cloud Agent installed. Answer “Yes” to all by default. It takes around 30 seconds.

 root@ip-172-31-19-255:~# wget -O install_script_ubuntu.sh https://my.evimetry.com/portal/install_script_ubuntu.sh?6a063dfa450349cef1a1dbc0eacd2b75af6e84ce

–2017-09-15 07:04:20–  https://my.evimetry.com/portal/install_script_ubuntu.sh?6a063dfa450349cef1a1dbc0eacd2b75af6e84ce

Resolving my.evimetry.com (my.evimetry.com)… 104.237.142.195

Connecting to my.evimetry.com (my.evimetry.com)|104.237.142.195|:443… connected.

HTTP request sent, awaiting response… 200 OK

Length: 9865 (9.6K) [text/plain]

Saving to: ‘install_script_ubuntu.sh’

100%[==================================================================================>] 9,865       –.-K/s   in 0s

2017-09-15 07:04:21 (216 MB/s) – ‘install_script_ubuntu.sh’ saved [9865/9865]

root@ip-172-31-19-255:~# bash install_script_ubuntu.sh

##########################################################################################

Updating APT and Installing dependencies

##########################################################################################

Run sudo apt-get –yes update [Y/n/a]?

<<<SNIP>>>

##########################################################################################

Configuring agent config for cloud deployment.

##########################################################################################

Artifact:    evimetry.agent

Description: Agent Application for Evimetry Application Suite

Version:     3.0.1

Build:       1117

Build Date:  2017-07-17T08:39:06.891+1000

evimetry.agent start/running, process 4917

############################################################################################

Evimetry installed and started. Point your controller at 172.31.3.157.

Control service by stop/start/restart evimetry.agent

Logs are in /var/log/upstart/evimetry.agent.log

Configuration is in /etc/init/evimetry.agent.conf

############################################################################################

The final thing to setup is a port forward so that we can connect through to the Evidence Server. Unlike some cloud services, EC2 Instances sit on a private IP address, behind a firewall.

In the “Network & Security” section of the console, go to “Security Groups”. Recalling from the Instance that is was started in the Security Group “Launch Wizard 3”, edit the inbound rules of that security group.

18-EditSecurity

Create a rule forwarding the Evimetry Cloud Agent’s port (TCP 9982) to the Evidence Server.

19-AddPortForward

Verify Access to Evimetry Cloud Agent

At this point, the Evimetry cloud agent is ready to be used. Using the public IP of the VM (54.206.15.84), connect in using the Evimetry Controller.

20-ConnectController

The agent will appear in the controller’s fabric nodes view (note that the IP of the Evidence Server is showing a 172.X.X.X private IP address). Visible underneath it is its storage, and an Evimetry Repository, which is located on its internal storage. We will acquire our images into this Repository.

21-ViewCloudAgent

Acquiring the image

We now go back to the EVIDENCE Security Domain, and access the “Elastic Block Store” | “Snapshots” section. Be sure to filter the view to “Private Snapshot” as it won’t be visible in the default setting.

The snapshot from the SUSPECT security domain will now be visible (check the Snapshot ID matches). We now transform the image into a Volume, which can then be added to a running instance in much the same way we plug removable storage into a computer. First, right click on the Snapshot and select “Create Volume”.

22-CreateVolume

In the volume creation form, where we create the evidence storage instance, we choose the Availability Zone of ap-southeast-2c”. Make sure that you choose this zone as the instance where the Volume is created, and then click on “Create Volume” to create the volume.

23-CreateVolumeDetails

Note the volume ID, of the new Volume, which is vol-097e59361bd515f78. Follow the link.

24-VolumeCreated

Now we can attach the volume to our Evidence Server. Go to the “Elastic Block Store” | “Volumes” area and, noting the Volume ID, select Attach Volume.

25-AttachVolume

 

Recalling that the instance ID of our Evidence Server is i-0af9148e32f37b8ac, attach the Volume as a virtual disk.

26-AttachVolumeDetails

 

Refreshing the Cloud Agent instance listed in the Evimetry Controller  now shows the disk attached to the agent as /dev/xvdf . Note that the newly attached disk is locked against mounting and writing. Right click on the disk and select Acquire.

27-AcquireVolume

The acquisition settings dialog will appear. Select a full linear acquisition of the attached drive, and add the Repository on the Storage Server as the container location. Give the Image a name using your standard image naming scheme, and document the original Volume ID and Instance ID associated with this image. Then click OK.

28-AcquireVolumeDetails

Acquisition is now underway. The screenshot below shows an acquisition using the “Provisioned IOPS SSD” as Volume storage, which proceeds at around 90MB/s, constrained by the storage of the infrastructure. Our testing shows that using “General Purpose SSD’s” as storage gives a trickling rate of around 10MB/s (that’s 4x slower than USB2!). A future post will focus on scaling this speed.

29-Acquiring

When the acquisition (including verification) completes, Evimetry will display a completion dialog.

30-AcquisitionCompleted

We then transfer the image locally to the lab using Evimetry, by flipping to the “Images” tab of the Controller, and right clicking on the newly created image.

31-TransferImage

After choosing the destination, the image downloads locally.

32-TransferImageDetails

Conclusion

This post has described a methodology for acquiring storage in the EC2 cloud. Using EC2 Snapshots in conjunction with Snapshot Sharing enables one to quickly Collect copies of Target storage. Acquisition can then be undertaken in the Cloud, so that the evidence is protected by a hash at the earliest opportunity, while minimising the amount of data that needs to be copied.

In future posts, I will follow up on how virtual disk selection affects the speed of acquisition; how to acquire volatile memory in AWS; and how to undertake analysis in the cloud.


Updated slides: Accelerating your forensic & incident response workflow

Screen Shot 2017-08-17 at 10.32.58 am

Late last year I had the pleasure of attending the F3 conference in Gloucestershire, UK. It is quite unlike any other digital forensics conference I have ever been to; a community run, practitioner focused, 2 day conference situated in a stately manor in the English countryside. I can thoroughly recommend it.

I had the opportunity to present an updated version of my presentation: “Accelerating your forensic & incident response workflow: the case for a new standard in forensic imaging”. The slide deck is available for download here.

 


Call for participation – AFF4 Working Group meeting at DFRWS 2017 USA

The Advanced Forensic Format 4 Working Group (AFF4 WG) is calling for interested parties to join the second working group meeting, to be co-located at the DFRWS Conference 2017, in Austin, TX.

Originally proposed in 2009 by Michael Cohen, Simson Garfinkel, and Bradley Schatz, the AFF4 forensic container enables new approaches to forensics, unparalleled forensic acquisition speeds and more accurate representation of evidence. The AFF4 WG has recently released v1.0 of the AFF4 Standard, including canonical images, specification, and open source libraries for implementers. Current AFF4 implementations include Rekall, Evimetry, Sleuth Kit, Volatility and GRR.

For more information, please see the working group mailing list, or contact Bradley Schatz or Michael Cohen.

Co-Chair: Dr Bradley L Schatz, Schatz Forensic/Evimetry, [ bradley <at> schatzforensic <dot> com ]
Co-Chair: Dr Michael Cohen, Google, [ scudette <at> google <dot> com ]

AFF4 working group mailing list: https://groups.google.com/forum/#!forum/aff4-wg


Compiling Sleuth Kit with AFF4 support on MacOS

We recently contributed patches to the Sleuth Kit to read AFF4 images. While we are waiting for those to be pulled into the main distribution, the following recipe should suffice for compiling a stand alone copy on MacOS.

Dependencies
The following dependencies are needed to compile libAFF4 on OSX. I use MacPorts, and the corresponding packages i needed to install are:

ossp-uuid
zlib
snappy
raptor2
google-glog
pcrexx
* tclap (missing *.pc file – place in /opt/local/lib/pkgconfig/)

Clone and compile LibAFF4 (C/C++)

Use the following to clone the current release of libaff4, configure it, and install.

git clone https://github.com/google/aff4.git
cd aff4
git submodule update –init third_party/gtest
cd third_party/gtest
git reset –hard
cd ../..
./autogen.sh
./configure CC=clang CXX=clang++ CXXFLAGS=”-std=c++11 -stdlib=libc++ -O2 -g0 -I/opt/local/include” LDFLAGS=”-stdlib=libc++ -L/opt/local/lib”
make
sudo make install

Clone and compile the Sleuth Kit

Use the following to compile the sleuthkit with libaff4 support.

git clone https://github.com/blschatz/sleuthkit.git
cd sleuthkit/
git checkout release-4.4
autoreconf –force –install –verbose
./configure
make
sudo make install

 


Evimetry v3 Released: Remote volatile memory support

We recently released Evimetry 3, the newest release of our revolutionary approach to forensic acquisition and analysis.

Whats new?

The big news is that we now support remote volatile memory acquisition. This means that in addition to being able to acquire remote disks, you can now acquire the volatile memory of live Windows, MacOS, and Linux hosts.  We primarily support Windows XP and above (x86 and x64) and OSX Mountain Lion and above (x64). The coverage for Linux memory acquisition is limited to 64 bit Intel machines where the kmem driver is enabled.

Get straight to analysis.

In addition to acquiring the physical memory, we also acquire and store the entry points needed to find the kernel page tables and base kernel data structures. The benefit of this is that time-consuming scanning for these entry points (which are fundamental to further analysis) can be bypassed getting you to analysing evidence sooner.

We have developed patches to the leading volatile memory analysis frameworks, Volatility and Rekall, to support reading these images, and the patches for Volatility have been contributed to the main Volatility project on GitHub.

Picture1

Acquire faster.

We take full advantage of Evimetry’s advanced compression to transport memory over the network at maximal rates. The effects of latency, a killer of network performance over long distance links, can be negated by pushing our networked evidence storage agents into the same network as the suspect computer.

Ready for digital evidence at wire speed?

If you would like to try these features, get in touch to organise an evaluation licence.

 


Sleuth Kit support for the AFF4 Standard v1.0 Released

I am pleased to announce the availability of both a set of patches to the Sleuth Kit and an open source C/C++ implementation for reading AFF4 Standard v1.0 disk images. Last week the AFF4 Standard v1.0 was released by Bradley Schatz (Evimetry) and Michael Cohen (Google) .

Screen Shot 2016-10-24 at 3.48.55 pm

Originally proposed in 2009 by Michael Cohen, Simson Garfinkel, and Bradley Schatz, the AFF4 forensic container enables new approaches to forensics, unparalleled forensic acquisition speeds and more accurate representation of evidence. These are enabled through next-generation forensic image features such as storage virtualisation, arbitrary metadata, and partial, non-linear and discontiguous images. The standard is the culmination of research spanning 6 years and 4 scientifically peer reviewed papers.

The release of these is a significant step forwards to the wider adoption of the format, enabling a large portion of the open source forensic toolchain to access AFF4 forensic images, and commercial implementers the ability to support reading the format by integration of a single unencumbered library.

The patches to the SleuthKit were contributed by Schatz Forensic (Evimetry), while the C/C++ library was originally developed by Michael Cohen (Google), with AFF4 Standard v1.0 support added by Schatz Forensic.

This release follows the release last week of the AFF4 Standard v.1.0 and a Python reference implementation (reader), and the release of Evimetry Community Edition, a freely licensed subset of the AFF4-based forensic tool.

For more information on the AFF4, attend the webcast “AFF4: The New Standard in Forensic Image Format, and Why You Should Care”, given by Bradley Schatz, in association with SANS, on 17 April 2017.

Implementers and interested parties are invited to join the AFF4 working group at aff4@googlegroups.com .


Introducing Evimetry Community Edition

Evimetry Community Edition provides a subset of the Evimetry system for free. The purpose of this is to grow the AFF4 ecosystem, firstly by providing a pain free path for Evimetry licensees to provide AFF4 images to non-licensees. Secondly, we wanted to provide practitioners, researchers and educators a freely available implementation of the AFF4 standard v1.0 which can be used to gain familiarity with the format. Schatz Forensic, the creators of Evimetry, drove the standardisation effort behind the AFF4 Standard v1.0.

With the Community Licenced Evimetry Controller, you can create Linear AFF4 Images on your Windows based analysis system, verify the integrity of AFF4 images, and convert between AFF4, E01/EWF and Raw images. You can also mount AFF4 images as virtual disks and analyse with your preferred forensic tools.

Using the Community Licenced Evimetry Filesystem Bridge, you can access entire repositories of AFF4 images as virtual raw files, enabling straightforward consumption with your existing forensic toolkit.

The release of Evimetry Community Edition coincides with the release by Schatz Forensic of open source implementations of the AFF4 format, patches to the Sleuth Kit supporting AFF4 images, and the release of the AFF4 Standard v1.0.

To gain access to the initial release of Evimetry Community Edition, email us at info@evimetry.com .


AFF4 Standard v1.0 Released

Today marks the release of the Advanced Forensic Format 4 (AFF4) Standard v1.0.

Originally proposed in 2009 by Michael Cohen, Simson Garfinkel, and Bradley Schatz, the AFF4 forensic container enables new approaches to forensics, unparalleled forensic acquisition speeds and more accurate representation of evidence. These are enabled through next-generation forensic image features such as storage virtualisation, arbitrary metadata, and partial, non-linear and discontiguous images. The standard is the culmination of research spanning 8 years and 4 scientifically peer reviewed papers.

Bradley Schatz (Evimetry) and Michael Cohen (Google) have collaborated to make freely available:
a set of canonical reference images which serve as ground truth for the format; and
• an explanatory specification document describing the format in detail; and
• a Python AFF4  reference implementation capable of reading the format.

This release of a standard specification for the file format is a milestone towards the wider adoption of the format, providing implementers an unambiguous and straightforward path to implementation. The release of the AFF4 Standard coincides with the limited release of Evimetry Community Edition, a freely licensed subset of the AFF4 based forensic tool, and in the coming days, a C++ implementation and patches to the Sleuth Kit, and support for Volatility and Rekall.

The standard specification and reference images are available at [1], the python implementation at [2], and aff4.org [3] becoming the central point of publication.

Implementers and interested parties are invited to join the AFF4 Working Group mailing list [4], and/or contact Bradley Schatz or Michael Cohen.

Contact:
Bradley Schatz ( bradley@evimetry.com )
Michael Cohen (scudette@google.com )

[1] https://github.com/aff4
[2] https://github.com/google/aff4
[3] http://www.aff4.org/
[4] https://groups.google.com/d/forum/aff4-wg


AFF4: The new standard in forensic imaging and why you should care

At this year’s Open Source Digital Forensics Conference (OSDFCon 2016) I presented an update on the AFF4 standardisation effort. For the conference we unveiled a significant milestone: support for consuming Evimetry produced AFF4 forensic images with the Sleuth Kit.

While users of Evimetry are able to exploit the benefits afforded by AFF4 seamlessly with their regular forensic tools, we believe that native support for the format across both opensource and commercial tools will accelerate forensic workflow even further.

The screenshot below demonstrates a non-linear partial physical image (containing only the allocated blocks from the target disk) being interpreted by the SleuthKit.

Screen Shot 2016-10-24 at 3.48.55 pm

We will be releasing patches for libaff4 (C++) and Sleuth Kit shortly.

My slides for the seminar are below.


Accelerating forensic and incident response workflow: AusCERT 2016 Slides

Existing forensic image formats are a bottleneck in the multi-core era: The slides from my recent presentation on accelerating forensic & incident response workflow at the AusCERT 2016 Conference. This summarises the research behind Evimetry Wirespeed.