File retrieval from a crashed drive

[piscator]

A bit of background on the ~$ smartctl -a command:

http://en.wikipedia.org/wiki/S.M.A.R.T.



What GParted looks like on a disk with lots partitions of Windows (ntfs) and Linux (ext4, swap) on it. Any drive on your computer will be simpler:

[klr]

Yes, I followed up on your suggestion a couple of days ago. The drive(s) were indeed running in IDE emulation mode. I switched to AHCI and made some other change connected with that. Unfortunately that did not fix anything. I contacted the mobo manufacturer's customer supprt, but have not heard back yet.

...

I have an old laptop with Windows XP. It originally game with Vista installed, but to get XP up and running, I needed to switch from AHCI to IDE emulation, and it has stayed that way ever since, even with Linux Mint 64 now also installed. I doubt Linux would complain if I went back into the BIOS and reversed the setting, and I wouldn't have expected it to help in your case either.

Have you looked at System Rescue CD?

http://www.sysresccd.org/SystemRescueCd_Homepage

I don't know if it has or does anything more than you've already tried. I've never really had cause to use it in anger. I back up often, and to multiple backup disks.

[piscator]

The point was to rule it out in any case. Since he noted the defunct drive shows up in both the Knoppix baitware and live Mint CD, and the 2009 mobo has UEFI problems, which necessitated a legacy install off CD.

I wanted to rescue Hermit's data before he installed Linux. But he installed Mint, and now recovering the crap on the old disk probably isn't worth the trouble at this point. He may take it up later.

[Hermit]

Have you looked at System Rescue CD?

Yes. That's quite a toolbox. Last night I revved up gParted/gpart. The data recovery menu item announced that it has to do a complete disk scan and that this will take a very long time. So I went to bed. This morning a message had appeared that gpart could not find a recognisable file system, which is no surprise because gParted had the file system listed as "unknown".

I have a few things to do today, but this evening I'll try some of the other programs out, like Partimage, TestDisk and PhotoRec. I need to find one that does not rely on the disk being mounted, a file system found or a sound MBR. Preferably it will not need to create partitions or write to the disk in any way at all. TestDisk does not look suitable for that. I'll try it if other programs won't do the trick. Partimage is said to work at block level. Maybe that means it does not need a mounted disk, a recognisable file system, but the "Part" part of its name seems to imply that it might need to see a partition. I'll find out tonight.

Oh, and thanks for your comment.

I wanted to rescue Hermit's data before he installed Linux. But he installed Mint, and now recovering the crap on the old disk probably isn't worth the trouble at this point. He may take it up later.

Linux has been installed on an entirely new disk. Nothing has been written to the original one since it refused to boot due to "one or several unrecoverable sectors". In fact, unless I am doing something like finding the path to it or try a data retrieval program on it, both the power and the data connections are removed because I want to minimise the time it keeps spinning. I figure that every time it spins up more sectors may become unrecoverable. In short, I don't know how installing Mint on a physically separate disk can adversely affect the chances of recovering data from the crashed one. Nevertheless, you have spent quite some time advising me, and I do appreciate that.

To be honest, the data recovery itself is not critical. My backup disk contains all but the last few weeks of what was written to it over the past six years plus all data files from the previous box I used, the one before that and selected bits from boxes reaching right back to the first one I bought 25 or 26 years ago. I iz bower bird writ large. Altogether there might be one or two dozen image files that I might lose, and the renaming of one or two hundred others. There are also several documentaries I downloaded, but they can be retrieved, mostly from youtube .... and porn sites, of course.

If I had known how much time I'd spend on this project, I probably would not even have bothered starting on it, but now it has become a means for me to get familiar with Linux, so I'll keep going with it. Linux is everything that M$ Win# increasingly is not. The amount of rubbish Win 8 includes by default is unbelievable, as is the amount of limits it places on what you can do.

Learning about computers has always been a byproduct of trying to accomplish a particular task for me. Initially I learnt by looking things up in hefty hardcover manuals that came with programs. They were gradually replaced by manuals on CDROM. Now it's Google for the win. This approach has its limits. For instance, Soon after I ought my first digital camera I would have liked to devise a tool with which I can automate the renaming of several thousand happy snaps in this uniform format: YYYY_MM_DD_HH_MM_SS, with the particular data for each extracted from the EXIF. That would have required me to learn a suitable programming language, meaning hundreds of hours of fairly structured and boring study. I'm too lazy for that. Never mind, though. Apparently someone else has spared me the effort by now, though the free version of it is probably crippleware, and I have not looked up what the full version would cost or if it can do everything I want it to.

Anyway, enough rambling for now.

[piscator]

My cameras seem to default to that anyway, but a spreadsheet could do that for you if you get or write a script* that extracts your meta to a CSV file, which you can then do anything with.

I think some may be reading this thread thinking how complicated it all is, but we're blending several topics into one discussion. I was hoping we could just do one thing at a time, but that takes forever.
Realistically, the drive is probably toast anyway. Unless you can recover or rewrite the data on the bad sectors somewhere else, there's not a lot of hope in economically reestablishing the data that describes how everything else is splattered over the platters.
Maybe if you go to enough jihad sites and threaten enough heads of state, your government would extract your data for you free of charge? That's how it works here...


Protip: Open a terminal. RtClick inthe terminal or find "Preferences" in the menu. "Edit Profile"...In the profile settings, you can enable scrolling, change behavior for a big output, and tell it how many lines to make scrollable. "Unlimited" works for me.
You can also change fonts and colors, in case you can't live without green ascii on a black field.





*A script is a group of commands, run in order. Aka "Batch file". The commands can be terminal commands, or a program written in a scripting language. Terminal commands are a good place to start scripting.

[Hermit]

Unless you can recover or rewrite the data on the bad sectors somewhere else, there's not a lot of hope in economically reestablishing the data that describes how everything else is splattered over the platters.

That is exactly what Runtime Software offers to do. It says it retrieves everything it can, backfills unrecoverable sectors with zeros and copies the lot to a destination I can specify. Trouble is, you have to pay for it. My suspicion is that the company took some bog standard Linux code, perhaps something like Partimage and did little more than add a GUI to it . I'm OK with making do with the console and learning to drive by command line.

Which makes it impossible to only talk about data retrieval, of course. The thread necessarily becomes a crash course for noobs to Linux as well. Sure, there's plenty to learn, but I anticipate feeling mighty spiffed if I finally retrieve succeed retrieving as much from the wrecked disk - even if the percentage of retrievable stuff turns out to be very low - and was able to turn my back on M$ controlling bloatware without getting captured by an even more monopolistic and avaricious bunch of cunts.

[Hermit]

Progress. Sort of.

Installed photorec on healthy drive and set it to search for all files on the crashed one and to copy them to a directory I named "Recover". It promptly announced that it was doing a first pass reading the disk. In six minutes its report stated that it had found and recovered 31874 files and the process will continue for about another four hours. That's the good news. The bad news is that although the hard disk activity light on the front panel of the computer is permanently lit the details of how many files have been found/recovered has not been updated after the initial burst. The only change being reported is the estimated time to completion. 45 minutes from starting it has blown out to 221 hours and keeps rising. I'll let it run for the rest of the afternoon, then abort the process, create a new directory (Recover2), run testdisk which will attempt to recover lost partitions and try again. If that does not work I'll check out dd and dd_recover. If that fails as well I'll just drill some holes through the platters and dispose of the dead drive.

[Hermit]

A little more progress.

Letting photorec do its job directly from the crashed drive never got past recovering 31874 files or proceeding beyond the first 60000 sectors. After about eight hours I pulled the plug on it and set testdisk to copy the drive's entire content to the good disk in the form of an image file. I did not expect that to work because I read somewhere that testdisk needs to know what file system the drive it is to read has to be formatted with and some Linux command, the exact expression of which I cannot now remember, listed it as "unknown". Nothing else worked so far, though, so I though may as well try and see what happens.

What happened was that it worked. testdisk had no problem at all figuring out that the file system was NTFS and started beavering away at the copy process. It took about six or eight hours to finish the job, but no problem. It just ran in the background while I dickered around on the interwebses. Job done, I revved photorec up once again, but set it to plough through the image file saved in the Recover2 directory. Yet again it bogged down about 62000 sectors into the job and yet again the estimated time to completion blew out into the three digit hours territory an kept rising. I was about to abort that process once more, but then thought I'll just let it run and catch some sleep. That turned out to be a good idea. When I woke up I found that the program had unbogged itself, recovered half a million files and the estimated time to completion had shrunk to three hours. At this rate there will be slightly north of a million recovered files after a bit over nine hours of retrieval.

This presents a problem: I am not going to sift through a million files to find the ones I am after. *.img, *.png and other graphic files are easy enough to scan. They can be made to show up as thumbnail images. as for the rest, forget it unless the file creation date has been preserved and the files can be listed in that order. The fact that none of them retained their original file names at this stage is of no help either. Also, I wonder how many of them are actually just fragments.

I have no idea if photorec simply terminates after completing pass 1, goes on to do a second pass or proceeds to stitch files saved across more than one sector together. In principle the latter ought to be doable. Each recovered sector is bound to have information in it telling the computer where to find preceding and following sectors of a long file are, and the first one of them, if recovered ought to contain the name the file has been originally saved as. Whatever, I'll just wait and see what happens next.

Be sure to read the next exciting and nailbitingly suspenseful episode of "A Blindfolded Patsy's Adventures in Digitalland" whenver or if ever it gets posted.

[Hermit]

And more progress.

photorec retrieved more stuff than I expected, and better yet, it successfully chained as many files stored over more than one sector up as recoverable sectors allowed. Moreover, it preserved the file format as well as their creation or modification date of most of them, so I know how to filter them in order to make it easier to find the ones I want to keep. That's the good news. To begin with, all *.dll and *.exe files can safely be deleted, which will reduce the clutter by quite a lot. That's the good news.

The bad news is that firstly photocec was unable to retrieve the original file name. Secondly, all retrieved files begin with "f" followed by a nine digit number. Thirdly, neither could it indicate which original directories they came from. Fourthly, I now have over a million recovered files stored in 2081 directories, and of course not all of them will be in functioning order. *.txt files in particular do not appear in their expected format. They show up as if opened in Notepad, file headers and footers and all. Fifthly, my 2TB drive has less than 10% of free space left.

The last problem is easily solved by moving the image file testdisk created and the more than a million files photorec recovered to my 3TB USB backup drive. As for the "rest", filtering and reinstating the files I am after does not seem as impossibly time consuming after all. I'll just have to work out exactly how to filter out what I want (mainly by file type and storage date) and how to persuade whatever file manager I'll finish up using to look at all the 2081 subdirectories they are stored in rather than one at a time.

Truly riveting stuff, whut?

To be continued

[piscator]

Good luck with that shit.

[Hermit]

Like I said, that should not be too difficult provided I can find a file manager that can filter files by a number of criteria. To start with I only need to locate data files that have been created or updated a few weeks before the drive spat the dummy, so I'll finish up with probably somewhere between a few dozen to maybe 300 files to check out if they are actually viewable. The files will end with .txt, .doc, .123, .mov, .avi, .mov and a few others.

Apart from randomly looking at a few of the covered ones I have not been doing much concerning the recovery process in the past few days. The recovery and file extraction processes took almost two days. Then it turned out that they filled my new drive to about 85%, so I decided to copy the proceeds to my 3T USB3 external backup unit. Unfortunately the desktop only has USB2 ports, so that part of the exercise took another almost two days. It finished earlier today. I thought I'd just leave the desktop chugging alone unimpeded and use the laptop instead. A PCI USB3 card is on my shopping list now along with another Western Digital 3 or 5 TB USB drive.

Fortunately none of the stuff I am trying to retrieve is time-critical, so there's no need to rush things. In other good news my random checks revealed more functioning video files than I expected to find. Not bad at all, considering the number od sectors some of them would occupy, though some others are definitely cactus, as are quite a few graphic files, such as thumbprints that were stored in just a single sector.

Anyway, I quite enjoy doing that sort of stuff once I get started on it. That's an advantage of inclining a little toward the CDO side of personality traits.

[piscator]

Yo Hermano, Midnight Commander File Manager might come in handy if your files are in some sort of archive format, as it's probably quite a bit more powerful than the default. Plus, it's terminal-based.

install with:
~$ sudo apt-get install mc

invoke with:
~$ mc
or
~$ sudo mc (if you need to be root)

Fullscreen the terminal on an alternate desktop. It should take you just a few minutes to pick up with this tutorial:

http://www.trembath.co.za/mctutorial.html

[Hermit]

Ah, I installed that a few days ago. Not into hand holding, is it? The tutorial comes in ever so handy. Thanks.

Meanwhile I have discovered that Office Libre's Calc opens Lotus *.123 spreadsheet files. Well, sort of. Graphs and macros are lost, but at least you can view and play with the cell contents. No version of Micro$soft's Excel does any of that. Eventually I'll be reinstalling the Lotus suite anyway, though. Wine or some emulator will hopefully enable me to do that.

[Hermit]

Anyway, here's a summary of what I learnt about recovering files from my crashed, unbootable and apparently unreadable hard drive using Linux. Step 1: Download and install two (quite small) programs. In terminal that takes one command line each. sudo aptget install x, where x is testdisk and photorec respectively. Step 2: Run each program in turn. testdisk creates an image file of the crashed drive and names it image.dd. photorec picks out all the files and file fragments it can find. For my 1TB drive that took about 8 or 10 hours. Fortunately both programs run completely unattended. Step 3: Use a file program that sorts and filters the retrieved stuff according to criteria that suit you best and copy them across to wherever you want them to live.

After some trial and error, that is what my drive recovery process boiled down to. Pretty simple, really. At this stage I have not gone through step 3, but picking on files fairly randomly, it has become obvious to me that a lot of files were retrieved totally intact, some were retrieved without the formatting that the originating word processer or whatever added to them (that is, text files are retrieved in pure ASCII form), some files that were stored across two or more sectors could not be stitched back together, and very inconveniently, all files have lost the names I or some program gave them when they were originally created or modified. The names are replaced with a leading "f" followed by a number of digits and finishing with .XXX.

Anyway, despite times where I felt exasperated and frustrated, I enjoyed the process of searching and discovery. Fringe benefit, and this is not a small one, is that is the kick in the butt I needed to start learning a version of Linux, which will eventually enable me to finally jettison Micro$oft's Windoze OS. It's been rather bloaty for years now and that is getting worse. More annoyingly is that it is rapidly becoming simultaneously more nannying, domineering and restrictive in what it lets you do.

[piscator]

You can run DOS or old Windows in VBox, which you can also run from the command line once you know what's going on. VBox 5.0 is coming out soon, though it may not represent much of an improvement on older hardware.

Lotus is an IBM product and IBM is or was one of the corporate sponsors of LibreOffice, which used to be OpenOffice and was started by Sun, which may explain things.

As to MS, they aren't as bad as Apple and I don't get all the hate for Win 8.1.