Thanks for your help... I dont have access to the computer now but I'll let
you know if it works out when I do.
On Mon, Sep 30, 2024, 1:16 PM Rusty Carruth via PLUG-discuss <
plug-discuss@lists.phxlinux.org> wrote:
> Oops, you are correct, the uniq command should have -w 34 list.of.files
> not -w list.of.files. Sorry! (here's what I'd typed and what I should
> have cut/pasted:
>
> root@rusty-MS-7851:/backups1/backup_system_v2# uniq -c -d -w 34
> sorted.new_filesA.md5|less ; wc -l sorted.new_filesA.md5
> 42279 sorted.new_filesA.md5
> root@rusty-MS-7851:/backups1/backup_system_v2# uniq -c -d -w 34
> sorted.new_filesA.md5
>
> sorry again!)
>
>
> Also, if you want to get a list of files and their MD5 sums from 'higher
> up' in the directory tree, just change the starting directory in your
> find command to that higher up location. However, you might need to run
> the entire find and md5sum sequence as root, if the directories (and
> files) you care about don't have read permission for you. (so, to find
> ALL files everywhere on your computer, change the ~ to /. You'll
> certainly get lots of permission denied errors if you do that as
> yourself and not root. But starting at / will traverse ALL directories
> on your computer, including /dev, and others you probably don't care
> about. There are some useful options to find (like, don't go to a
> different filesystem) you might want to use, see man page for find to
> find them ;-)
>
> On 9/30/24 07:05, Michael via PLUG-discuss wrote:
> > thank you so much! After running it I find it only finds the duplicates
> in
> > ~. I need to find the duplicates across all the directories under home.
> > after looking at the man file and searching for recu it seems it recurses
> > by default unless I am reading it wrong.
> > I tried the uniq command but:
> >
> > uniq -c -d -w list.of.files
> > uniq: list.of.files: invalid number of bytes to compare
> >
> > isn't uniq used to find the differences between two files? I have
> > a very rudimentary understanding of linux so I'm sure I'm wrong
> >
> > all the files in list.of.files are invisible files. (prefaced with a
> > period))
> > and isn't there a way to sort things depending on their column (column1
> > md5sum, column2 file name)
> >
> > On Mon, Sep 30, 2024 at 2:56 AM Rusty Carruth via PLUG-discuss <
> > plug-discuss@lists.phxlinux.org> wrote:
> >
> >> On 9/28/24 21:06, Michael via PLUG-discuss wrote:
> >>> About a year ago I messed up by accidently copying a folder with other
> >>> folders into another folder. I'm running out of room and need to find
> >> that
> >>> directory tree and get rid of it. All I know for certain is that it is
> >>> somewhere in my home directory. I THINK it is my pictures directory
> with
> >>> ARW files.
> >>> chatgpt told me to use fdupes but it told me to use an exclude option
> >>> (which I found out it doesn't have) to avoid config files (and I was
> >>> planning on adding to that as I discovered other stuff I didn't want).
> >> then
> >>> it told me to use find but I got an error which leads me to believe it
> >>> doesn't know what it's talking about!
> >>> coul;d someone help me out?
> >>>
> >> First, someone said you need to run updatedb before running find. No,
> >> sorry, updatedb is for using locate, not find. Find actively walks the
> >> directory tree. Locate searches the text (I think) database built by
> >> updatedb.
> >>
> >>
> >> Ok, now to answer the question. I've got a similar situation, but in
> >> spades. Every time I did a backup, I did an entire copy of everything,
> >> so I've got ... oh, 10, 20, 30 copies of many things. I'm working on
> >> scripts to help reduce that, but for now doing it somewhat manually, I
> >> suggest the following command:
> >>
> >>
> >> cd (the directory of interest, possibly your home dir) ; find . -type f
> >> -print0 | xargs -0 md5sum | sort > list.of.files
> >>
> >> this will create a list of files, sorted by their md5sum. If you want
> >> to be lazy and not search that file for duplicate md5sums, consider
> >> uniq. Like this:
> >>
> >> uniq -c -d -w list.of.files
> >>
> >>
> >> This will print the list of files which are duplicates. For example,
> >> out of a list of 42,279 files in a certain directory on my computer,
> >> here's the result:
> >>
> >> 2 73d249df037f6e63022e5cfa8d0c959b
> >>
> >>
> _files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160321-223138.png
> >> 5 9b162ac35214691461cc0f0104fb91ce
> >> _files/melissa/Documents/EPHESUS/Office Stuff/SPD/SPD SUMMER 2016
> (1).pdf
> >> 3 b396af67f2cd75658397efd878a01fb8
> >> _files/dads_zipdisks/2003-1/CLASS at VBC Sp-03/CLASS BKUP - Music
> >> Reading & Sight Singing Class/C & D Major & Minor Scales & Chords.mct
> >> 2 cd83094e0c4aeb9128806b5168444578
> >>
> >>
> _files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160318-222051.png
> >> 2 d1a5a1bec046cc85a3a3fd53a8d5be86
> >>
> >>
> _files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160410-145331.png
> >> 2 fa681c54a2bd7cfa590ddb8cf6ca1cea
> >>
> >>
> _files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160312-113340.png
> >>
> >> Originally the _files directory had MANY duplicates, now I've managed to
> >> get that down to the above list...
> >>
> >> Anyway, there you go. Happy scripting.
> >>
> >> ---------------------------------------------------
> >> PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org
> >> To subscribe, unsubscribe, or to change your mail settings:
> >> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >>
> >
> >
> > ---------------------------------------------------
> > PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org
> > To subscribe, unsubscribe, or to change your mail settings:
> > https://lists.phxlinux.org/mailman/listinfo/plug-discuss
> ---------------------------------------------------
> PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>
---------------------------------------------------
PLUG-discuss mailing list:
PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss
(text/plain)