Dissecting Video and Audio for Further Examination
Mar 11, 2021 23:52:41 GMT
ag47, trolljegeren, and 2 more like this
Post by somerandomuser on Mar 11, 2021 23:52:41 GMT
Often we may think that powerful and expensive software is needed for dissecting video footage, but this is not the case at least software needing to be expensive. Powerful software is often times free and opensource, which can be modified to suit your own purposes, for example my video/audio machine is a Linux (Ubuntu 18.04) with ffmpeg, ShotCut, GIMP, Audacity and a few other standard Linux packages all of which is completely free. All this same software is free in Windows as well and on my Windows machine I keep ShotCut and GIMP, mainly because the main machine has a little more computing power which helps greatly when working on videos. Video tends to chew up some cpu cycles for sure in processessing and there are times where files get quite large or take a long time to render.
So let's start out by talking about camera's, mainly digital cameras as they are everywhere. These cameras on average shoot video between 24 and 30 frames per second (FPS). Just for a comparison the human eye processes about 24-48 frames per second. Imagine that every second you are taking in 24-48 fps to comprehend a scene, so we are going to be talking about a lot of images for a one minute video. Also lower FPS we may find that we are missing some of the more subtle things movement might be jerky or the subject moves very quickly, we’ve all seen this I am sure. Finding out the particular phone/camera used is very important since some cameras can shoot 60 even 120 frames per second which would make the video in “slow motion”.
www.nytimes.com/2018/04/17/smarter-living/beginners-guide-phone-video.html
datagenetics.com/blog/may12018/index.html
A word on lossless formats for audio and video some file formats actually cause small loses in audio quality or video quality thus leading to “artifacts”. These artifacts are not actually in the video this is common on YouTube because most videos are converted into a format for faster display of the video.
Audio
lifehacker.com/does-bitrate-really-make-a-difference-in-my-music-5810575
Video
www.archivalworks.com/blog/lossless-video-compression
Artifacts
en.wikipedia.org/wiki/Compression_artifact
Visual Artifacts
en.wikipedia.org/wiki/Visual_artifact
Along the lines of figuring out cameras always check metadata, there are small bits of information in pictures, videos and other files which sometimes give us GPS or equipment used to capture the media you are working with. I always check files, usually using "exiftool" an opensource software on GitHub and works on both Windows, Linux, and OSX. This metadata can further glean information from the media which you are examining, but should you trust it? Absolutely not as it can be tampered with and fraudulent, therefore looking at everything in totality is important.
I always suggest and try to use metadata in order to verify the camera used and make sure that the video is within the specs of that camera. Denote that social media sites now scrub metadata from images/video and other such media perhaps not all of them but many do; So if you are investigating a video which a friend has taken always get a copy of the original. I will include a tutorial on mobile device data acquisition since more than likely we will have to deal with these at some point. While this is not a forensic investigation the importance of ensuring that data has not been changed is important.
Let’s take the following scenario, we find an interesting video on Youtube and we want to further analyze it, at first it sounds a bit daunting. How the heck do we get that video? It’s not too hard, we use “youtube-dl” another free and opensource software from GitHub and works in Windows, Linux, and OSX. There are several different sites and programs which can download videos from YT, but I personally have not used them, but there are other options. A small word that on Windows/OSX/Linux you can easily install this package using “pip” a python package installer/manager (only for python packages/libraries). I really should mention that youtube-dl is command line based, either cmd, powershell or a terminal (don’t be scared of the typing it’ll be okay). Often for command line interface based programs you will find that the –help or even the GitHub repository has fairly decent documentation with it so there isn’t really anything to be worried about (you won’t goof anything up). If say we are downloading using youtube-dl, the command is rather straight forward using:
youtube-dl www.youtube.com/watch?v=rY_ewcSaRho
github.com/ytdl-org/youtube-dl
Of course replace with the video in the example above with one you are wanting to download. For myself I will usually have a “Videos” folder where in I issue my commands as being somewhat organized will help down the road. I also suggest having a flash drive for this work, as three seconds of a subject being on video means 90 pictures if we are extracting 30 fps, it gets to be a lot to keep track of if one is not organized. (Flash drives and external drives are not expensive maybe 4 bucks for 16GB and 1TB external is 50-70 bucks, even at WalMart).
From there we need to extract frames, remember where we said that knowing the particular device be it from asking or checking metadata is important? This is where that comes into play we can extract frames at pretty much any speed we want, for example we can do 24 fps (stock for most phone cameras)or 120fps. At the very least we want to match the original video, I wouldn’t want to extract all the videos at say 120 fps because you will find a lot of repetition in the frames. Personally I don’t like to push the fps too far usually no more than 6fps more than what it is, and yes you will have reptition, so be ready to see a subject standing the same for a couple of frames. Usually for every 14 seconds using 6 extra FPS or 25% during extraction you add 1 second of video time when you take the sequence of photos and make it a video again. A lot of this has been trial and error on my part and I have found that the 25% rule seems to work.
Something I really want to touch on is using a “lossless” picture and audio formats. The ubiquitous jpg(JPEG) are compressed so a bit of detail tends to be missing, PNG is “lossless”. Expect PNG’s to be between 200kb – 5 MB per picture so that removable media (Flash Drive/External HD) is very helpful.
Extracting frames from a video is rather straight forward and this is where FFMPEG comes into play. FFMPEG is a powerful piece of software used for converting video, audio, stream and recording, it’s also free, I should note that there are several other libraries for doing other video/audio edits, also for the most part it’s command line based. I know command line, but it’s really not as bad as it seems for example let’s say we are extracting at 30 fps, the subject comes into view at 46 seconds and maybe is on for 14 seconds, here is how we would do it:
ffmpeg -ss 00:00:46 -t 00:00:14 -i TX_DM_Vid.webm out%05d.png
(Breakdown of Command: ffmpeg calls ffmpeg/-ss is the start position/-t is the duration we are extracting frames for/ -i File_Name_here.ext (the filename that you are looking at)/ out%5d.png (the output format and it will number them sequentially from 1 – wherever it finishes(this syntax is in BASH, PowerShell is a bit different syntax (check out Google if you are stuck let me know glad to help.))
FFMPEG works with nearly every format of video so sometimes you might end up with odd ball extensions or formats, generally don’t worry.
Project Site: ffmpeg.org/
Documentation: ffmpeg.org/ffmpeg.html
I always like to scroll through the frames individually, sometimes you can see little things. Flipping back and forth between frames is also a good idea, small movements sometimes are not easy to discern, but I have found that focusing on quadrants or the body and cycling through the frames a few times picking up on the little things really helps.
A word on audio, I use Audacity which is a free and opensource audio editor. Most videos have sound and sometimes interesting things can be found in the audio. You can use YouTube-DL for this as well doing the following:
youtube-dl -F URL_HERE
youtube] Setting language [youtube] HRIF4_WzU1w: Downloading webpage [youtube] HRIF4_WzU1w: Downloading video info webpage [youtube] HRIF4_WzU1w: Extracting video information [info] Available formats for HRIF4_WzU1w: format code extension resolution note 171 webm audio only DASH webm audio , audio@ 48k (worst) 140 m4a audio only DASH audio , audio@128k 160 mp4 192p DASH video 133 mp4 240p DASH video 134 mp4 360p DASH video 135 mp4 480p DASH video 17 3gp 176x144 36 3gp 320x240 5 flv 400x240 43 webm 640x360 18 mp4 640x360 (best)
You will see a small table appear where you can choose your format, generally “audio only” (140). In addition there are sites which do copy the audio only and let you download it for free. If you already know the format then use:
youtube-dl -x –audio-format mp3 URL_HERE
Note: mp3 is an example you can use other formats such as wav but try to go for better quality formats as not to lose information.
Depending on how you want to clean up and enhance the audio Google is going to be your best friend. For example say we want to remove vocals and there is no track title (audacity needs this to remove vocals in “Karaoke”) you can add a title (Edit > Metadata) or find which frequencies are human speech. (It may differ depending on the audio and proximity/pitch of the people.) (Doing it through frequencies is the tough manual way.) I suggest that one tries for the best audio quality within reason and always use headphones. Let’s say that an audio player doesn’t support the format, well that is not an issue as ffmpeg can convert audio to other formats.
I suggest that people have a specific folder for the video and frames extracted as sometimes it can become quite a bit of frames (3 seconds is 90 pictures if extracting at 30fps). Also always backup extracted images before touching them as you want to allow others to be able to replicate your processes. Take notes when doing all of this when dealing with so many images one can quickly get confused, make notes of commands used, programs and frames you find interesting during when manually scrolling through them.
A few parting words
One thing I found that helped me was creating helper scripts if say I have multiple videos where in some I only want audio and maybe some where I want audio and video. I also use a VirtualMachine (VM) with an attached USB or external hard drive for storage of really important and interesting data. I think that the only reason I segment my machines is because I don’t want or need all the tooling all the time and have learned to build the software packages around what that particular VM is going to help me do. While I might be using Linux for my audio/video analysis if one is more comfortable with Windows, Microsoft has free VirtualMachines from Windows 7 -10 (Developer Tools) in addition to Server 2012 – 2016 (probably 2022 although doubtful this is needed) in the evaluation center.
developer.microsoft.com/en-us/microsoft-edge/tools/vms/
Something else when working on audio there is quite a few resources which can help one decipher what they are looking at say on a spectograph. One will find many articles on bioacoustics and often analysis is done via software (generally programming in R (I don’t R or CRAN currently I only knew it from DFIR)) but if say we had a recorder out in the field we can filter out everything but specific calls. Also R allows for graphing and other similar functions if I get more into audio analysis then I may start creating helper scripts which of course I would share.
GIMP, ShotCut, Audacity and SonicVisualizer are although in my toolbox although there are many good tutorials which would be a much better and in depth of a resource than if I was to cover them here. Obligatory Disclaimer Audio/Video work is not a strong point also I may post the source code for my helper scripts usually in python if that there is interest.
So let's start out by talking about camera's, mainly digital cameras as they are everywhere. These cameras on average shoot video between 24 and 30 frames per second (FPS). Just for a comparison the human eye processes about 24-48 frames per second. Imagine that every second you are taking in 24-48 fps to comprehend a scene, so we are going to be talking about a lot of images for a one minute video. Also lower FPS we may find that we are missing some of the more subtle things movement might be jerky or the subject moves very quickly, we’ve all seen this I am sure. Finding out the particular phone/camera used is very important since some cameras can shoot 60 even 120 frames per second which would make the video in “slow motion”.
www.nytimes.com/2018/04/17/smarter-living/beginners-guide-phone-video.html
datagenetics.com/blog/may12018/index.html
A word on lossless formats for audio and video some file formats actually cause small loses in audio quality or video quality thus leading to “artifacts”. These artifacts are not actually in the video this is common on YouTube because most videos are converted into a format for faster display of the video.
Audio
lifehacker.com/does-bitrate-really-make-a-difference-in-my-music-5810575
Video
www.archivalworks.com/blog/lossless-video-compression
Artifacts
en.wikipedia.org/wiki/Compression_artifact
Visual Artifacts
en.wikipedia.org/wiki/Visual_artifact
Along the lines of figuring out cameras always check metadata, there are small bits of information in pictures, videos and other files which sometimes give us GPS or equipment used to capture the media you are working with. I always check files, usually using "exiftool" an opensource software on GitHub and works on both Windows, Linux, and OSX. This metadata can further glean information from the media which you are examining, but should you trust it? Absolutely not as it can be tampered with and fraudulent, therefore looking at everything in totality is important.
I always suggest and try to use metadata in order to verify the camera used and make sure that the video is within the specs of that camera. Denote that social media sites now scrub metadata from images/video and other such media perhaps not all of them but many do; So if you are investigating a video which a friend has taken always get a copy of the original. I will include a tutorial on mobile device data acquisition since more than likely we will have to deal with these at some point. While this is not a forensic investigation the importance of ensuring that data has not been changed is important.
Let’s take the following scenario, we find an interesting video on Youtube and we want to further analyze it, at first it sounds a bit daunting. How the heck do we get that video? It’s not too hard, we use “youtube-dl” another free and opensource software from GitHub and works in Windows, Linux, and OSX. There are several different sites and programs which can download videos from YT, but I personally have not used them, but there are other options. A small word that on Windows/OSX/Linux you can easily install this package using “pip” a python package installer/manager (only for python packages/libraries). I really should mention that youtube-dl is command line based, either cmd, powershell or a terminal (don’t be scared of the typing it’ll be okay). Often for command line interface based programs you will find that the –help or even the GitHub repository has fairly decent documentation with it so there isn’t really anything to be worried about (you won’t goof anything up). If say we are downloading using youtube-dl, the command is rather straight forward using:
youtube-dl www.youtube.com/watch?v=rY_ewcSaRho
github.com/ytdl-org/youtube-dl
Of course replace with the video in the example above with one you are wanting to download. For myself I will usually have a “Videos” folder where in I issue my commands as being somewhat organized will help down the road. I also suggest having a flash drive for this work, as three seconds of a subject being on video means 90 pictures if we are extracting 30 fps, it gets to be a lot to keep track of if one is not organized. (Flash drives and external drives are not expensive maybe 4 bucks for 16GB and 1TB external is 50-70 bucks, even at WalMart).
From there we need to extract frames, remember where we said that knowing the particular device be it from asking or checking metadata is important? This is where that comes into play we can extract frames at pretty much any speed we want, for example we can do 24 fps (stock for most phone cameras)or 120fps. At the very least we want to match the original video, I wouldn’t want to extract all the videos at say 120 fps because you will find a lot of repetition in the frames. Personally I don’t like to push the fps too far usually no more than 6fps more than what it is, and yes you will have reptition, so be ready to see a subject standing the same for a couple of frames. Usually for every 14 seconds using 6 extra FPS or 25% during extraction you add 1 second of video time when you take the sequence of photos and make it a video again. A lot of this has been trial and error on my part and I have found that the 25% rule seems to work.
Something I really want to touch on is using a “lossless” picture and audio formats. The ubiquitous jpg(JPEG) are compressed so a bit of detail tends to be missing, PNG is “lossless”. Expect PNG’s to be between 200kb – 5 MB per picture so that removable media (Flash Drive/External HD) is very helpful.
Extracting frames from a video is rather straight forward and this is where FFMPEG comes into play. FFMPEG is a powerful piece of software used for converting video, audio, stream and recording, it’s also free, I should note that there are several other libraries for doing other video/audio edits, also for the most part it’s command line based. I know command line, but it’s really not as bad as it seems for example let’s say we are extracting at 30 fps, the subject comes into view at 46 seconds and maybe is on for 14 seconds, here is how we would do it:
ffmpeg -ss 00:00:46 -t 00:00:14 -i TX_DM_Vid.webm out%05d.png
(Breakdown of Command: ffmpeg calls ffmpeg/-ss is the start position/-t is the duration we are extracting frames for/ -i File_Name_here.ext (the filename that you are looking at)/ out%5d.png (the output format and it will number them sequentially from 1 – wherever it finishes(this syntax is in BASH, PowerShell is a bit different syntax (check out Google if you are stuck let me know glad to help.))
FFMPEG works with nearly every format of video so sometimes you might end up with odd ball extensions or formats, generally don’t worry.
Project Site: ffmpeg.org/
Documentation: ffmpeg.org/ffmpeg.html
I always like to scroll through the frames individually, sometimes you can see little things. Flipping back and forth between frames is also a good idea, small movements sometimes are not easy to discern, but I have found that focusing on quadrants or the body and cycling through the frames a few times picking up on the little things really helps.
A word on audio, I use Audacity which is a free and opensource audio editor. Most videos have sound and sometimes interesting things can be found in the audio. You can use YouTube-DL for this as well doing the following:
youtube-dl -F URL_HERE
youtube] Setting language [youtube] HRIF4_WzU1w: Downloading webpage [youtube] HRIF4_WzU1w: Downloading video info webpage [youtube] HRIF4_WzU1w: Extracting video information [info] Available formats for HRIF4_WzU1w: format code extension resolution note 171 webm audio only DASH webm audio , audio@ 48k (worst) 140 m4a audio only DASH audio , audio@128k 160 mp4 192p DASH video 133 mp4 240p DASH video 134 mp4 360p DASH video 135 mp4 480p DASH video 17 3gp 176x144 36 3gp 320x240 5 flv 400x240 43 webm 640x360 18 mp4 640x360 (best)
You will see a small table appear where you can choose your format, generally “audio only” (140). In addition there are sites which do copy the audio only and let you download it for free. If you already know the format then use:
youtube-dl -x –audio-format mp3 URL_HERE
Note: mp3 is an example you can use other formats such as wav but try to go for better quality formats as not to lose information.
Depending on how you want to clean up and enhance the audio Google is going to be your best friend. For example say we want to remove vocals and there is no track title (audacity needs this to remove vocals in “Karaoke”) you can add a title (Edit > Metadata) or find which frequencies are human speech. (It may differ depending on the audio and proximity/pitch of the people.) (Doing it through frequencies is the tough manual way.) I suggest that one tries for the best audio quality within reason and always use headphones. Let’s say that an audio player doesn’t support the format, well that is not an issue as ffmpeg can convert audio to other formats.
I suggest that people have a specific folder for the video and frames extracted as sometimes it can become quite a bit of frames (3 seconds is 90 pictures if extracting at 30fps). Also always backup extracted images before touching them as you want to allow others to be able to replicate your processes. Take notes when doing all of this when dealing with so many images one can quickly get confused, make notes of commands used, programs and frames you find interesting during when manually scrolling through them.
A few parting words
One thing I found that helped me was creating helper scripts if say I have multiple videos where in some I only want audio and maybe some where I want audio and video. I also use a VirtualMachine (VM) with an attached USB or external hard drive for storage of really important and interesting data. I think that the only reason I segment my machines is because I don’t want or need all the tooling all the time and have learned to build the software packages around what that particular VM is going to help me do. While I might be using Linux for my audio/video analysis if one is more comfortable with Windows, Microsoft has free VirtualMachines from Windows 7 -10 (Developer Tools) in addition to Server 2012 – 2016 (probably 2022 although doubtful this is needed) in the evaluation center.
developer.microsoft.com/en-us/microsoft-edge/tools/vms/
Something else when working on audio there is quite a few resources which can help one decipher what they are looking at say on a spectograph. One will find many articles on bioacoustics and often analysis is done via software (generally programming in R (I don’t R or CRAN currently I only knew it from DFIR)) but if say we had a recorder out in the field we can filter out everything but specific calls. Also R allows for graphing and other similar functions if I get more into audio analysis then I may start creating helper scripts which of course I would share.
GIMP, ShotCut, Audacity and SonicVisualizer are although in my toolbox although there are many good tutorials which would be a much better and in depth of a resource than if I was to cover them here. Obligatory Disclaimer Audio/Video work is not a strong point also I may post the source code for my helper scripts usually in python if that there is interest.