Computer Associates, Alternate Data Streams, and why you should be concerned. And what you might be able to do about it.
January 22, 2007. Now that my issue is resolved I've reorganized and consolidated this page for readability.
Description of Alternate Data Streams (ADS)
Articles about ADS
Tools for working with ADS
My Computer Associates Tech Support Rant
Updates on successfully removing the Computer Associates ADS from my computer
After several years of use I recently uninstalled Computer Associates ETrust Anti-virus when my license expired and (because I try to support companies that provide Free versions) I switched to AVG's
paid version on that computer. They also have a free version
, which I use on another computer. Just after that I heard about an Alternate Data Stream (ADS) scanner named LADS. When I ran it on one of my drives I was shocked to find that I had over 17,000 hidden objects that ETrust had apparently placed there as part of it's virus scanning procedure. They're "Alternate Data Streams", and one was associated with every file on my NTFS partitions. CA has confirmed that they contain a hash of the file so that the AV didn't need to rescan if the hash hasn't changed. The Streams all had CA_INOCULATEIT as part of the name, so it's obvious where they came from. Because of the potential abuse of Alternate Data Streams by malware, any decent malware scanner needs to scan each of them in addition to the normal files. All 17,000 plus in my case, doubling the time and CPU cycles necessary to do a complete scan. And I have 2 NTFS drives, the second one contained even more ADS.
That got me looking around for a solution. Naturally I contacted Computer Associates, thinking that surely they'd have a method of removing their [and only
their] tags. Well, I tried to contact them. As is usually the case, getting a coherent answer from Tech Support was not very productive. It's only fair to note that after most of this page was done, thanks to an outside party I was put in touch with a manager at Computer Associates who was very helpful, and tracked down a CA solution for me. Details are in the updates
I want to specifically thank Stefana Muller, the Product Manager for eTrust Consumer Software for all her help in getting this straightened out. She was helpful, knowledgeable and prompt in getting answers and fixes. To some extent all's well that ends well.
But since I originally had trouble getting any help from Computer Associates I started doing my own research into ADS and how I could fix my problem.
After some poking around the internet and with the assistance of some of the participants of the GRC
newsgroups (particularly Mark V. who knows far more than I do about NT file systems and related stuff) I put together some links to information on Alternate Data Streams and some programs that will find them, and in a few cases, remove them.
An Alternate Data Stream is sort of
a file. All files and folders on NTFS are Streams of one type or another. The primary file is actually an UN-named
Stream. Alternate Data Streams are named streams and have to be associated with conventional files. To access them the fully qualified
name must be used. To open a text ADS in Notepad the command would be "notepad FileName:StreamName.txt". ADS can not exist on FAT formatted drives, only on NTFS drives. Alternate Data Streams can be any size and any type of file/data, and there can be multiple Streams associated with a single file. And any type of file/data can be in a stream and it can be associated with any type of file. So a program file could be hidden in a stream associated with (as an example) an image file. Microsoft created them to provide compatibility with files from the MAC OS. The problem is that they are hidden from normal methods of file inspection in Windows. They won't show up in file lists in Explorer or if you do a DIR from the command prompt, and Windows doesn't include their size in any typical calculations that it does. And of course any time that you have files that are hidden from Windows and it's users, you have the potential for malware to take advantage of it. There are reports of some BHO's using ADS to store their information and additional files. Viruslist.com
briefly mentions one example of a virus using Alternate Data Streams to spread across multiple drives once an infection is in place, there are others out there too. There's even a remote exploit of some software recently reported by Secunia
that takes advantage of a vulnerability in the way ADS is handled. I don't know if it's possible to use the same type of exploit in other circumstances, I imagine we'll find out eventually.
It's possible to have a zero byte file that has multiple streams of various types associated with it. So you can have a file that appears to take up no space other than it's file listing, when in fact it might have several large and hidden ADS files associated with it. If you find that you have unexplained zero byte files in unusual places it would certainly be a good idea to use one of the stream scanners listed further down this page to make sure that they aren't just 'place holders' for Alternate Data Streams.
While ADS can only exist on an NTFS drive, they can be accessed, read, and (apparently) run from a non-NTFS drive. So a program on a FAT32 drive can use an ADS as long as it is running in an OS that can access NTFS drives. ADS are apparently also backed up, at least when using MS's Backup programs. This means that while you can't copy a file with ADS to a non-NTFS drive and have the ADS survive, it appears that you could run a backup to a FAT drive, then restore it to NTFS and the ADS would survive. Probably. Some compression programs may preserve ADS, but I tested WinZip and IZarc and they don't, even if the zipped file remains on an NTFS drive. That doesn't mean that it's impossible for a virus/trojan writer to find a way to compress a file and it's associated ADS and send it by email. If your compression program isn't "aware" of ADS though, it probably wouldn't unzip them even if you were foolish enough to unzip the file itself, they'd simply disappear. I think, I'm certainly no expert. The good news is that it's apparently impossible for ADS to be included in a normal file attachment and be sent through email, it will be stripped off.
Keep in mind that those ADS won't show up in a file listing, aren't accounted for in file size calculations and can't be seen from any native Windows application. Although I did find out that CHKDSK
includes them in used and free space calculations, but not in the file numbers. And while many operating system files in Windows 2000 and XP are protected by System File Protection, it's possible to add ADS to those protected files and not have them detected. So you can have ADS in critical Windows system folders and Windows won't make any noise about it at all. Fortunately programs that are hidden in ADS can be seen when they're running if you know where to look and what to look for. It's far from obvious though.
Not only can all that add up in terms of (hidden) used space on your drive, but Windows needs to keep track of each of these ADS, even when Windows won't show them to you. That can take up a lot of extra space in the Master File Table
or MFT. And as they come and go it can cause fragmentation both on the drive and in the MFT, which can have a significant effect on the computer's operation. ADS can also be associated with folders, not just the files in them. The good news is that an ADS will not run just because the associated file is run, they have to be started specifically. But that's easy for malware to do once they've managed to get them on your drive.
There are a couple of issues regarding the ADS related programs I've found. Many of them won't list ADS associated with folders [or directories if you prefer], just those associated with files. Only a few of them will show ADS associated with some of the System files, "like System Volume Information\tracking.log". LADS will list them, and Streams will clean them though. As far as I know none of them will clean an ADS that has the Read Only attribute set. The attribute appears to be inherited from the file that the ADS is associated with. Some of those files may need that attribute, so if you change it to allow the deletion of the ADS, then it needs to be changed back afterwards. On the files I tested changing the Read Only attribute on the parent file allowed the ADS to be deleted. But since ETrust has an ADS associated with every
file on my computer, including all the system files (many of which need to be Read Only), that means a lot of extra work for manual deletion. I've also found one ADS scanner, SFind from Foundstone.com
, (go to Resources, Free Tools, Forensic Toolkit) that loops and starts over when it comes across a file that it can't access, which makes it fairly useless, at least on my computer. In my case it's a test file that my Anti-virus locks, and apparently that throws SFind for a loop. Literally.
Frank Heyne has some good information on ADS in his FAQ
Kaspersky used to use ADS to store hash data for files, but they stopped using them in early 2006. According to the press release
it wasn't for security reasons, it was to speed up the removal of their software since it took a lot of time to remove all the ADS. However they also have an earlier article
about the danger of virus writers using ADS for malicious purposes. And one problem with AV companies using them to store data is that the sheer number of ADS could hide malicious files using the same technique. I know that I would certainly have trouble picking anything unusual out of the mass of Computer Associates ADS on my computer.
According to Microsoft's KnowledgeBase Article Q101353
which was last reviewed in November of 2006 (as of this writing) Windows NT is inconsistent in it's support for Alternate Data Streams, so detectors, cleaners and other ADS related activities may not always do what's expected. I'm not sure how much of that may have carried over into NT's descendants, Windows 2000 and XP. KB105763
is more recent, although less recently reviewed, and it indicates that at least some aspects may not have changed.
Microsoft's KB Article 319300
mentions that The Windows 2000 Content Indexing Server adds an ADS that contains a thumbnail to image files.
And Microsoft's articles 883260
have information about how IE stores origin data on files downloaded from the internet. This is what generates those warning when you try to execute a file that was downloaded using IE. There's only indirect references to it, but if you know what they're talking about you can see that the data is stored in an ADS. And other articles on the internet support that. So if you have a file that is bugging you with warnings every time you use it, you can use any of the ADS removal methods listed further down this page
to get rid of the warning.
has an excellent article on Alternate Data Streams, with links to programs and more information.
has a good article titled The Dark Side of NTFS (Microsoft's Scarlet Letter)
with links to programs and more information.
A more recently updated page with lots of information is at bleepingcomputer.com
. It even tells how
to remove them. But none of the methods are practical for system files unfortunately, and all would be extremely tedious for the number of ADS that I'm dealing with.
Before doing any ADS cleaning be sure that you have a good back up available. A full image is advised.
Remember that if you restore the backed up files, it will also restore the ADS if the backup program is streams aware. I use DriveSnapShot
for my backups, and I've tested it, it does
preserve ADS if the file is copied back to an NTFS drive, which is actually a Good Thing. Once you succeed in cleaning the ADS you'll want to defragment your drive. I defragged mine before I cleaned it and the defragger told me it didn't need to run again afterwards. It seems to me that if it's only going to be run once, it would be better to run defrag after
the cleaning when the ADS space will be absorbed in the major defrag process. I also suggest running CHKDSK before and after cleaning the ADS.
I've tested the first 6 of these and they all work exactly as advertised. I haven't tried the others but I'm listing them for the sake of completeness.
LADS from Frank Heyne Software
will list all your ADS files. LADS is a command line program, but LADS will not delete ADS files from the drive. LADS does have a filtering option that you can use to not show specific ADS. In fact he uses the Computer Associates ADS in an example. Using this command line for his program will scan the C: drive, including subfolders, and will not list any ADS with the string "CA_INOCULATEIT" in the name:
"LADS C:\ /s /xCA_INOCULATEIT"
. This makes it particularly useful for getting a count on specific types of ADS on a drive.
ADSSpy from Merijn is a GUI based program and it will delete Streams. You can use it to find all ADS, then select the ones you want to remove by selecting them from the list. There are several options like Select All and Invert Selection available on a right click menu.
is from Mark Russinovich of Sysinternals.com, now part of Microsoft. It's a command line tool and can be used to find and delete streams.
If you go about half way down Frank Heyne's FAQ
you'll see a link to a Microsoft
page that provides a download link near the top for NTFSext.exe. Download the file and follow the instructions on Frank's page [or other sites if you search] on how to add a tab to the file Properties dialog that will show you Stream information on files. It won't help you find files with streams, but it will help you get information about them once you know which files to look at. I've added some setup details
at the bottom of this page.
Computer Associates had an older tool that would delete the Streams created by an earlier version of their Anti-virus program. And only their Streams, something that helped considerably and made it fairly simple to use. While I was testing it I found a couple of minor issues with the tool, which they fixed. However they seem to have removed it [as of November 2007] since that page is not found. So for now you have to use the older version, and you have to jump through a few hoops and multiple downloads to get it. There's details in my additional notes
section, including an explanation of what circumstances would result in the ADSs to begin with. It was only an older version that would do it, and only if an option was manually enabled. And my excellent results from using this tool are in my last update
I haven't tried the programs below this point, some of them because they require an installation, and I prefer stand alone software for something like this.
requires an installation to run and looks very similar to Microsoft's NTFSext.exe
. According to the author's page it may even require lowering some of your security settings to run, which isn't something I normally recommend. I haven't tried it. But he has some very good information about ADS on his page. Stream Viewer will show Streams associated with folders, and it will remove streams. But you have to look at files one at a time.
requires an installation and lets you browse files and their streams in an interface similar to Windows Explorer. It does not delete ADS. But you can actually view the content of the ADS. That can also be done with NTFSext (see above) with a little extra work. Like Stream Viewer you can only deal with one file at a time.
has an ADS tool, but I haven't given them the information they want to allow me to download it. In my opinion they want too much information for a simple download of a basic tool and a write-up. I'm told that it's a simple GUI program that lists ADS on all or specified drives, and has no option to delete them. At least that was true for the version available in early 2005. I'm not going to jump through the hoops to see if it's changed.
is another command line tool for finding ADS. It does not have the capability to delete them.
Microsoft's Stream Viewer Shell Extension
StrmExt.dll is the critical file for the basic functions. That file needs to be removed from the extracted list of files, placed in the System32 folder and registered. Two other files may be required for additional functions, but I haven't tested that. Mark V provided some nice instructions
and comments which I've included in a separate text file.
Computer Associates Streams cleaner utility.
It seems that ETrust AV only added ADS if someone enabled "Incremental Scanning". This was a feature in a rather old version, probably around the year 2000. Apparently I thought that it seemed like a good idea at the time and turned it on, and I'm reasonably certain that ADS weren't mentioned anywhere. The Stream contains a time stamp, a scan result and the engine/signature version that was used for the scan. On the next scan the ADS was checked first and matched to the file information. If there was no change, the file itself didn't need to be scanned again. That was eventually dropped when the driver's cache was improved.
So they created a utility to delete the Streams. All the references I can find to it on their site are dated 2002 and 2003. It looks like at some point the tool was included as part of a new version of their software and must have run automagically after an upgrade, resulting in removal of most of the ADS from most users' computers. But they apparently failed to include it on the CD for version 7.0, so they put it on their web site, which means it's available for download. Update: The following section had been superseded by the updated CA delstrm.exe mentioned in my Tools section. I left it in in case the other link quit working and because some of these links provide additional information. And as of November 2007 the other link is dead, so these links are where you need to go. Unfortunately these links are to the old version that has some issues.
It's in an [apparently] proprietary zip file with an extension of CAZ. Download the "QO40477.CAZ" file listed at the bottom of QO40477
and save it to a folder on your computer. Then download CAZIPXP
which is the extractor. Instructions on how to use it are on that page. It must be used at a command prompt and there are no error messages if you make a mistake like mistyping the name of something. Once you have delstrm.exe extracted you can use the example at FAQ287184
to run the cleaner. The cleaner also needs to be used at a command prompt. The command "delstrm -V C:\ CA_INOCULATEIT" would remove most of the Computer Associates Streams on the C drive. I say "most
" because it will not remove Streams from Read Only files. And it won't state that in the screen display that results, it will simply say that no Streams were found in those files. There's also no log output, so if data scrolls off the top of the command prompt window it's lost. FAQ282104
contains a bit more information about the CA cleaner and their use of Streams. The Read Only issue had been fixed, but the fixed version is now missing, so Read Only files will need to be dealt with differently.
My rant about Computer Associates Tech Support
I spent a few weeks jumping through hoops and filling out online forms trying to find someone who knew something. Unfortunately that's become pretty standard with a lot of big companies and outsourced tech support. I had no luck at all and just went in circles for a couple of weeks until someone in a newsgroup gave me an email address for the 'CA Product Manager - eTrust Consumer Software'. I'll update the page if I get a response.
Update (later on) December 5, 2006
I received a response from the Product Manager within about 4 1/2 hours, giving me what information she knew and saying that she'd do some research to find a solution. Before I'd even seen the first email, I got a follow-up with a link to a program that CA created to remove their Streams. It appears that the streams were used by a very old version of their program, and that a later version that I probably didn't install might have cleaned them auto-magically. I'll need to do some experimenting with their program to see how well it works, it runs from a command line and has limited instructions. I've asked if it's OK to post the link to it.
Update December 10, 2006
I've exchanged several emails with the Product Manager. Computer Associates does
have a tool (listed in the section above
) that will remove the Streams created by an earlier version of their ETrust Anti-virus program. I'm waiting to hear from someone involved with writing the software since I've found a couple of issues with it. See the notes below.
Update January 13, 2007
Where does the time go? Christmas, winter storms with power outages, work, life in general. Anyway, within a couple of days of the previous update Computer Associates had fixed their tool so that it will clean their Streams from Read Only files. It's now available on their support site without jumping through the hoops I described previously. I have a link to it in the section above
. I did some quick testing that verified this, but haven't had time to do a thorough trial on my System drive because I want to do backups, scans, analysis, then clean it, then do more scans and analysis. Hopefully soon. But to be fair to CA I wanted to add this update. I have to say that the response in this case was outstanding, far better than my previous experiences.
Update January 22, 2007
I spent a few hours last week cleaning my computer and taking notes. If you're specifically interested in removing the Computer Associates ADS, then their tool will now get all but one or two of them, which is acceptable to me. The remaining ADS is associated with a file that I can't even see without quite a bit of work, "System Volume Information\tracking.log". What I did was run CHKDSK to verify that the drive was OK and find out how many files there were. Then I ran LADS and ADSSpy to get the number of ADS and the number of Computer Associates ADS. Then I defragged the drive. I ran the Computer Associates tool using the -V switch, which cleans their Streams. Then I ran CHKDSK, LADS and ADSSpy again for comparison. 17,324 of 17,325 Computer Associates Streams were removed, without touching the other 518 ADS on the drive. Another pass with Defrag said there was no need to run it. I'd recommend running defrag after the cleaning instead of before so that any slack space resulting from ADS removal is absorbed in a larger defrag. The small bits that result from the cleaning aren't enough for defrag to notice. Someday when I have time I'll try to remove that remaining Stream using Mark Russinovich's Streams.
It's interesting to note that based on my very limited experience here, CHKDSK does not count Streams in calculating the number of files on the disk. But it does count Streams when it's calculating how much space is used and free space. Here's the results of running CHKDSK just before and after removing the 17,234 Streams. Before is in Black, after is in Red. They're interleaved for easy comparison. I didn't include the lines that aren't relevant to this.
12361985 KB total disk space.
12361985 KB total disk space.
No change in total disk space.
16236 KB in 3570 indexes.
16236 KB in 3570 indexes.
No change in the number of indexes.
63872 KB occupied by the log file.
63872 KB occupied by the log file.
No change in space occupied by the log file.
115525 KB in use by the system.
115525 KB in use by the system.
No change in the amount disk space in use by the system.
4788828 KB in 46817 files.
4751444 KB in 46817 files.
A 37,384 KB difference in the amount of space in use, even though the number of files didn't change.
7441396 KB available on disk.
7478780 KB available on disk.
And a matching 37,384 KB difference in space available on the disk.
So Alternate Data Streams don't affect the number of files in use on a drive, they're counted as being part of the un-named stream file, but they do affect the amount of space used by those files. Only in CHKDSK though, if you look at File Properties the extra space used by the ADS isn't included by Windows. Removing ADSs from a file has no effect on the file size that Windows reports.
The good news is that I've learned a lot from all of this, and hopefully this page will help out others who may have questions about ADS, especially the ones created by an older version of ETrust.
Keep in mind that I am not responsible for any external sites linked to from my pages. They may look different to you, or even have effects on your browser or computer that are different than what I see due to different security settings and browsers. They could have also changed since I looked at them. To the best of my knowledge, they are all safe. But you surf at your own risk.
This document reflects the opinions of the author. This document is provided "as is" without any express or implied warranties. While every effort has been taken to ensure the accuracy of the information contained in this article, the author/maintainer and/or contributors assume(s) no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.
The only information that I collect is page hit counts. My web host Penguinhost.net keeps track of lots of things and makes the information available to me in pretty graphs and logs. I look at them occasionally, but there is no personally identifiable information there.
Validated by HTML Validator (based on Tidy)
This page was last updated Dec 10, 2009
Copyright 2006-2009 by Kevin