Just lost a hard disk drive - oh joy !

[GettingTooOld]

made a slip with dd, dd is a linux utility that lets you write disk images. Don't know it's real name, only it's nickname "disk destroyer". Instead of aiming for the USB thumb drive, it zapped the file onto the base of the second hard disk drive, destroying at least six months of everything and backups of years past. collecting some tatters now, and cannot shut anything down until that is done because it will never start up again. Plus, it won't let me connect an external hard drive to the computer right now. Only usb drives of which i don't have many. So have to choose very carefully what to try and save.

Such a tiny slip. Literally one single letter. The resulting destruction is awesome. Never to old to make a huge blunder.

[TriciaG]

Next time use Windows.

Kidding! Kidding!

[GettingTooOld]

Why, do you think windows can destroy a disk any better than I can ? I was plenty fast, or maybe you mean as a secret overnight task ?
wakey wakey rise and shine your commercial business is now been erased by an unrequested upgrade to windows whatever.
yes kidding, perhaps Ok everyone, you are all witnesses Tricia started the windows Vs Linux discussion

I have to try to either spend time choosing which child won't make it onto the usb lifeboat, or try to work out how to restore the partitions by gathering info before shutting down. ergh.

Gathering information seems to work ok. I find the dimensions of the old partitions in /sys/block/ drive id and it has text of size and start in files there, in the memory. I've typed them down so that I can make a new partition table the same size and shape as the old one. I hope. I'll still lose plenty of information though. Sigh.

[GettingTooOld]

I've managed to get back about 1/3 of the work I had.
I managed to get back one whole partition containing a lot of video that couldn't be replaced and other things. I'm relieved, but I still lost a huge amount. Huge. I might be able to get back some more, and I'm learning a lot about how to bullet-proof any new systems I do, to better protect them from my blazing stupidity reflexes. Ooops.

Oh, and I can still use the other hard drive in the laptop to run the system, there were copies of my favorite operating system on both, along with various other operating systems, so either one can start the laptop and so on. I definitely need a sacrificial partition at the start of each disk to guard against this sort of hazard.

[redrun]

My hands-on experience with this particular tool is limited (thankfully I haven't had to use it "for real"), but the 'testdisk' utility is designed for recovering from exactly this type of thing. It can rebuild that partition table using that information you noted, and help you get at the rest of the (surviving) files more easily.
You won't need the "download a live cd" bit, but here's a basic tutorial to start with:
https://linuxconfig.org/how-to-recover-partition-table-in-linux

Then of course, there's 'scalpel', which can look through even a hard drive without a partition table at all, and look for files matching certain parameters (JPEG or PNG image headers, .zip compression blocks, etc.) and copy them automatically elsewhere:
https://www.redhat.com/sysadmin/find-lost-files-scalpel
This only helps to the extent you have room, and I've found that it has quite a few false-positives, but... the more tools you have available to you, the better the chance you'll have what you need. Maybe you already knew both of these, but they're worth mentioning in case.

Sorry to hear about your drive, and I sure hope you're able to recover the rest!

[GettingTooOld]

My hands-on experience with this particular tool is limited (thankfully I haven't had to use it "for real"), but the 'testdisk' utility is designed for recovering from exactly this type of thing.[..]
You won't need the "download a live cd" bit,[...]

That one has been working very well for me. testdisk and fdisk and some mathematics and general knowledge. I got some information from testdisk, and fed it into fdisk. fdisk really had me getting my hopes up when I had entered the correct start and end cylinders from the live memory in what was left of the dying system and also testdisk too I think. There was a set of numbers that wasn't so fabulous and needed calculation for one partition. But when I put the right numbers into it, it went ahead and simulated the new parameters and then it said :

Code: Select all

Created a new partition 5 of type 'Linux' and of size 7.8 GiB.
Partition #5 contains a swap signature.
Do you want to remove the signature? [Y]es/[N]o: n

Created a new partition 6 of type 'Linux' and of size 43.1 GiB.
Partition #6 contains a ext3 signature.
Do you want to remove the signature? [Y]es/[N]o: n

Created a new partition 7 of type 'Linux' and of size 443.7 GiB.
Partition #7 contains a ext4 signature.
Do you want to remove the signature? [Y]es/[N]o: n

Partition #8 contains a ext3 signature.
Do you want to remove the signature? [Y]es/[N]o: n

Can you imagine how I felt right there. Doesn't get better than that, or at least I hope it does...

I didn't need a live CD as I have so many multiboot options to choose from, however, you MUST stay in the crashed system for as long as you can gathering information, because once you shut it down it will never come back, and it has a wealth of information in memory about where it came from. The city that has now been destroyed, on the planet that exploded, in the galaxy that was vacuumed away to make way for an expressway of some sort.

I saw and ran a bash script that grabbed numbers out of memory and wrote them to the console for you, that was pretty cool as it was easy to use. I already had the numbers by then, but this whole episode makes me think of how to make a system much better, and it's recovery much easier.

Also, it really didn't hurt having a dozen operating systems spread across two hard drives in the laptop. I even went and installed GIMP on a system I had not used much up till now and found it works a LOT better.

Reminds me of the Mad Hatters tea party, migrating whatever you can from one placement to the next on a regular basis. It has a lot of advantages as well as the obvious disadvantages.

Then of course, there's 'scalpel', which can look through even a hard drive without a partition table at all,[...]
This only helps to the extent you have room, and I've found that it has quite a few false-positives, but... the more tools you have available to you, the better the chance you'll have what you need.

Yes, well some serious storage is front and center on the shopping list once more. There's a minefield. Never get WD and it's other face, segate. There are so many bad choices and pain to experience with those brands. WD, if Only I had a dollar for every one of those that destroyed itself within a month of being NEW. OH, wait, I would recommend them if there's a pretty girl at the returns counter and someone shy is asking me how to approach them.

Maybe you already knew both of these, but they're worth mentioning in case.
Sorry to hear about your drive, and I sure hope you're able to recover the rest!

Thankyou, and on behalf of people who may remember this thread when their hard drive is destroyed through misadventure in the future, and refer to it for details, and I do mean a group of people which surely will include me, again, thank you for the helpful advice. I certainly did not know about scalpel.

More space to allow scalpel to do it's work, and study scalpel. That's the agenda. Luckily there is enough recovered so I already got back to work, with new enthusiasm, hoping to get work finished before it's all destroyed by the next disaster.

[GettingTooOld]

Then of course, there's 'scalpel', which can look through even a hard drive without a partition table at all, and look for files matching certain parameters (JPEG or PNG image headers, .zip compression blocks, etc.) and copy them automatically elsewhere:
https://www.redhat.com/sysadmin/find-lost-files-scalpel
This only helps to the extent you have room, and I've found that it has quite a few false-positives,

Finally able to assemble the gear required and run this for the first time. It required a glibc library, I downloaded a few of those, and the package itself a few versions, and then tested until I found a combination of those and a suitable selection of operating system. I formatted a Terabyte hard-drive (following a newly implemented pattern I thought up to make recovery easier. Use nice round numbers for the size of every partition and have a sacrificial partition at the start of the drive, right in the zone that a disk destroy command would target with a USB stick sized projectile.)

So it required descriptions of files, it has a few really basic ones. Ran it, it found 20 thousand pictures. Lots of false positives, lots of stuff I have no idea where it came from, really. There's like hundreds which when flicked through are like a very old version of grand theft auto or something which I have never played. It's obviously picked them out of a video, but where the video came from stumps me. Then there are others which are quite wide and perhaps 3 pixels high. They seem to be slivers of images that need stacking. Lots of false positives.

On the other hand, there are plenty of direct hits !!

They all have consecutive nonsensical numeric filenames now, but that's better than nothing.

No actual videos. No descriptions to give it.

The file identifiers are not a difficult thing to find, there are lists in the internet, even lists in my computer, there must be, because my computer knows how to identify file types. Getting those descriptions in a format that is ready-to-go however is something I have yet to find out. There must be some configuration files out there ready-made on the internet so it's not worth trying to write them one at a time myself. Search engines are not generally helpful creatures, so I thought to ask you for any ideas you may have from your 'limited' experience with it. asking a search engine is never as good as asking a person, the search engines are stumped, but I''ll persist.

[redrun]

Scalpel is filesystem-independent, which means it can find files of a given format within any filesystem... but it also doesn't look for metadata blocks and file names, unfortunately.

Many of the small or odd-shaped images you find might be web assets. I ended up using 'find'[1] to sort through scalpel's output and remove .jpg and .png files too small to be the photos we wanted kept. Similarly, I used 'grep'[2] to search for common patterns in the useless text files, to winnow those down.

[1]: For example, to list all .jpg files of less than 1 MB in size:

Code: Select all

find . -name "*.jpg" -size -1M

You can either keep the list of file names, removing them later with 'rm', or add '-delete' at the end to have 'find' delete matching files itself.

[2]: For example, weeding out some pesky INI files, of which some program seemed to have written hundreds:

Code: Select all

grep -rlIF "[IniVersion]"

You probably know these, but I always find a refresher useful.
'-F' searches the literal string rather than a regex pattern, if you'd rather not deal with escaping special characters for every search.
'-I' skips binary files that might happen to contain what you're searching for - I wanted to handle those separately, plus it speeds up the search.
'-l' will list only the file names, rather than also printing the matching text - this makes it easy to save the list of file names, for passing to 'rm' once you're sure you want them gone. There's also 'cut', but this was just one step simpler.
'-r' is, of course, to search recursively through all the files in sub-folders, rather than only in the current folder itself.

[GettingTooOld]

You can either keep the list of file names, removing them later with 'rm', or add '-delete' at the end to have 'find' delete matching files itself.

[2]: For example, weeding out some pesky INI files, of which some program seemed to have written hundreds:

Code: Select all

grep -rlIF "[IniVersion]"

You probably know these, but I always find a refresher useful.
'-F' searches the literal string rather than a regex pattern, if you'd rather not deal with escaping special characters for every search.
'-I' skips binary files that might happen to contain what you're searching for - I wanted to handle those separately, plus it speeds up the search.
'-l' will list only the file names, rather than also printing the matching text - this makes it easy to save the list of file names, for passing to 'rm' once you're sure you want them gone. There's also 'cut', but this was just one step simpler.
'-r' is, of course, to search recursively through all the files in sub-folders, rather than only in the current folder itself.

Find I'm very familiar with, use it all the time, forget whe I put things so often it's compulsory and I wish it could be used in real life. wouldn't that be good, recursive search of house and find all your socks again !

grep, I've used, but it's about time I brushed up.

unfortunantley it doesn't address the key issue, which is finding filetypes for scalpel to identify. Sorting them afterwards is a somewhat manual somewhat automated task and I'm fine with that, getting used to doing that already, before this started.

Here's irony, I can get it back, and find it all again online is easy, but I have lost the txt file I jotted down my one-liner which renamed all files matching a pattern like *.jpg into orderly numbered files with a nice prefix. It would get rid of all the tracking flienames like rivyu4gmfsv8w2uf.jpg and so on, and give a list of 00001.jpg 00002.jpg but now, that's all I got careful what you wish for eh? (scalpel outputs 001, 002, 003 for all pics) well, I think some compromise is ideal.

[edit] got it back ! from a more recent system...

Code: Select all

i='0'; for filename in *.jpg; do let "i=i+1"; mv $filename ${i}.jpg;echo $i; done

i='0'; this initialises a variable to use to count with, so you will have a total number at the end
for filename in *.jpg; this loops the directory & filename is a variable which has each one in turn
do let "i=i+1"; this counts 1+1 for each file renamed
mv $filename ${i}.jpg; this renames whatever is stored in filename variable to 000001.jpg etc
echo $i; this gives a total number renamed
done

so people can also change this to batch rename audiofiles where there is a mistaook in the tiple.

But I haven't seen a single movie file as yet, as I haven't got the identifiers. Hmm, on with the search. By the time this ends, I should be ok at data recovery. The only other immense nut to crack would be bringing back broken hardware hard drives, the ones that simply don't start up. That's an ideal a bit too big to hope for I'd say.

edit: found them, on the site you referred to..
off to find the right place to save them. Every time I turn off the system I'm using for that, it wipes out because it was in memory. Have to write it all down this time.

[GettingTooOld]

I made a 120GB image from the lost partition, and asked scalpel to go after too much it seems. It choked the 500GB partition it was on with 300Gb of unpacked found files. False positives galore. An interactive way to do it would be so much better given it's about 6 or 7 hours per attempt.

Better than not working though. Gotta be said. It's the best there is cause it's all there is.

[GettingTooOld]

the tool is able to bring back images, but not so much video, it only brings back short sections of video, 20? seconds at a time or so. It only brings back set sizes of chunks to interpret, and so it's whatever fits into a chunk. Have not seen any txt files as yet.

[loon]

Oh! - dd - so much power in such an innocent-sounding command. I haven't been there there, but I've managed to blow up disks in other ways. Take your time, get some sleep, etc.

"Now I am become dd, destroyer of worlds."

[GettingTooOld]

"Now I am become dd, destroyer of worlds."

That's exactly it. linux command line is the bomb. I hate how everything not-linux doesn't allow anything you want it to do. The customizations people have done to linux is incredible. Downside of all that power is all that power. But if you are using linux and think you're missing anything, you could always trash your harddrives whenever your business friends are having their compulsory windows upgrades, so you don't miss out on the pain and mayhem. yay windows. Oh, and you'll need to send copies of your hard drives to microsoft as well.

So far I can bring back at least some images, and now I have found out how to bring back text file contents.

Code: Select all

hexdump -C fileimagefragmentMadewithdd | grep -i clues

this will help you search for lines that contain the text 'clues' using hexdump, although the lines are of course short, so it won't find

Code: Select all

25fd4f30  6f 6e 20 74 68 65 20 69  6e 74 65 72 6e 65 74 20  |yon the inter cl|
265f3280  66 74 20 49 6e 74 65 72  6e 65 74 20 45 78 70 6c  |ues Internet Exp|
265f3200  39 30 37 32 22 7d 2c 22  69 6e 74 65 72 6e 65 74  |9042"},"mix n ma|

but will find

Code: Select all

266a1140  45 7c 49 6e 74 65 72 6e  65 74 20 45 78 70 6c 6f  |E2inclues plo|
2d513680  66 74 20 49 6e 74 65 72  6e 65 74 20 45 78 70 6c  |these clues6%|
2d5a5230  39 30 37 32 22 7d 2c 22  69 6e 74 65 72 6e 65 74  |more clues nt|

fragments of audio pop up when the images are played as vlc files, if they agree to be played. probably could split an audiofile and cat them into it.