thank you so much! After running it I find it only finds the duplicates in
~. I need to find the duplicates across all the directories under home.
after looking at the man file and searching for recu it seems it recurses
by default unless I am reading it wrong.
I tried the uniq command but:
uniq -c -d -w list.of.files
uniq: list.of.files: invalid number of bytes to compare
isn't uniq used to find the differences between two files? I have
a very rudimentary understanding of linux so I'm sure I'm wrong
all the files in list.of.files are invisible files. (prefaced with a
period))
and isn't there a way to sort things depending on their column (column1
md5sum, column2 file name)
On Mon, Sep 30, 2024 at 2:56 AM Rusty Carruth via PLUG-discuss <
plug-discuss@lists.phxlinux.org> wrote:
>
> On 9/28/24 21:06, Michael via PLUG-discuss wrote:
> > About a year ago I messed up by accidently copying a folder with other
> > folders into another folder. I'm running out of room and need to find
> that
> > directory tree and get rid of it. All I know for certain is that it is
> > somewhere in my home directory. I THINK it is my pictures directory with
> > ARW files.
> > chatgpt told me to use fdupes but it told me to use an exclude option
> > (which I found out it doesn't have) to avoid config files (and I was
> > planning on adding to that as I discovered other stuff I didn't want).
> then
> > it told me to use find but I got an error which leads me to believe it
> > doesn't know what it's talking about!
> > coul;d someone help me out?
> >
> First, someone said you need to run updatedb before running find. No,
> sorry, updatedb is for using locate, not find. Find actively walks the
> directory tree. Locate searches the text (I think) database built by
> updatedb.
>
>
> Ok, now to answer the question. I've got a similar situation, but in
> spades. Every time I did a backup, I did an entire copy of everything,
> so I've got ... oh, 10, 20, 30 copies of many things. I'm working on
> scripts to help reduce that, but for now doing it somewhat manually, I
> suggest the following command:
>
>
> cd (the directory of interest, possibly your home dir) ; find . -type f
> -print0 | xargs -0 md5sum | sort > list.of.files
>
> this will create a list of files, sorted by their md5sum. If you want
> to be lazy and not search that file for duplicate md5sums, consider
> uniq. Like this:
>
> uniq -c -d -w list.of.files
>
>
> This will print the list of files which are duplicates. For example,
> out of a list of 42,279 files in a certain directory on my computer,
> here's the result:
>
> 2 73d249df037f6e63022e5cfa8d0c959b
>
> _files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160321-223138.png
> 5 9b162ac35214691461cc0f0104fb91ce
> _files/melissa/Documents/EPHESUS/Office Stuff/SPD/SPD SUMMER 2016 (1).pdf
> 3 b396af67f2cd75658397efd878a01fb8
> _files/dads_zipdisks/2003-1/CLASS at VBC Sp-03/CLASS BKUP - Music
> Reading & Sight Singing Class/C & D Major & Minor Scales & Chords.mct
> 2 cd83094e0c4aeb9128806b5168444578
>
> _files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160318-222051.png
> 2 d1a5a1bec046cc85a3a3fd53a8d5be86
>
> _files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160410-145331.png
> 2 fa681c54a2bd7cfa590ddb8cf6ca1cea
>
> _files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160312-113340.png
>
> Originally the _files directory had MANY duplicates, now I've managed to
> get that down to the above list...
>
> Anyway, there you go. Happy scripting.
>
> ---------------------------------------------------
> PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>
--
:-)~MIKE~(-:
---------------------------------------------------
PLUG-discuss mailing list:
PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss
)%0A>%20and%20isn't%20there%20a%20way%20to%20sort%20things%20depending%20on%20their%20column%20(column1%0A>%20md5sum,%20column2%20file%20name)%0A>%20%0A>%20On%20Mon,%20Sep%2030,%202024%20at%202:56%E2%80%AFAM%20Rusty%20Carruth%20via%20PLUG-discuss%20<%0A>%20plug-discuss@lists.phxlinux.org>%20wrote:%0A>%20%0A>%20>%0A>%20>%20On%209/28/24%2021:06,%20Michael%20via%20PLUG-discuss%20wrote:%0A>%20>%20>%20About%20a%20year%20ago%20I%20messed%20up%20by%20accidently%20copying%20a%20folder%20%20with%20other%0A>%20>%20>%20folders%20into%20another%20folder.%20I'm%20running%20out%20of%20room%20and%20need%20to%20find%0A>%20>%20that%0A>%20>%20>%20directory%20tree%20and%20get%20rid%20of%20it.%20All%20I%20know%20for%20certain%20is%20that%20it%20is%0A>%20>%20>%20somewhere%20in%20my%20home%20directory.%20I%20THINK%20it%20is%20my%20pictures%20directory%20with%0A>%20>%20>%20ARW%20files.%0A>%20>%20>%20chatgpt%20told%20me%20to%20use%20fdupes%20but%20it%20told%20me%20to%20use%20an%20exclude%20option%0A>%20>%20>%20(which%20I%20found%20out%20it%20doesn't%20have)%20to%20avoid%20config%20files%20(and%20I%20was%0A>%20>%20>%20planning%20on%20adding%20to%20that%20as%20I%20discovered%20other%20stuff%20I%20didn't%20want).%0A>%20>%20then%0A>%20>%20>%20it%20told%20me%20to%20use%20find%20but%20I%20got%20an%20error%20which%20leads%20me%20to%20believe%20it%0A>%20>%20>%20doesn't%20know%20what%20it's%20talking%20about!%0A>%20>%20>%20coul;d%20someone%20help%20me%20out?%0A>%20>%20>%0A>%20>%20First,%20someone%20said%20you%20need%20to%20run%20updatedb%20before%20running%20find.%20%20No,%0A>%20>%20sorry,%20updatedb%20is%20for%20using%20locate,%20not%20find.%20%20Find%20actively%20walks%20the%0A>%20>%20directory%20tree.%20%20Locate%20searches%20the%20text%20(I%20think)%20database%20built%20by%0A>%20>%20updatedb.%0A>%20>%0A>%20>%0A>%20>%20Ok,%20now%20to%20answer%20the%20question.%20%20I've%20got%20a%20similar%20situation,%20but%20in%0A>%20>%20spades.%20%20Every%20time%20I%20did%20a%20backup,%20I%20did%20an%20entire%20copy%20of%20everything,%0A>%20>%20so%20I've%20got%20...%20oh,%2010,%2020,%2030%20copies%20of%20many%20things.%20I'm%20working%20on%0A>%20>%20scripts%20to%20help%20reduce%20that,%20but%20for%20now%20doing%20it%20somewhat%20manually,%20I%0A>%20>%20suggest%20the%20following%20command:%0A>%20>%0A>%20>%0A>%20>%20cd%20(the%20directory%20of%20interest,%20possibly%20your%20home%20dir)%20;%20find%20.%20-type%20f%0A>%20>%20-print0%20%7C%20xargs%20-0%20md5sum%20%7C%20sort%20>%20list.of.files%0A>%20>%0A>%20>%20this%20will%20create%20a%20list%20of%20files,%20sorted%20by%20their%20md5sum.%20%20If%20you%20want%0A>%20>%20to%20be%20lazy%20and%20not%20search%20that%20file%20for%20duplicate%20md5sums,%20consider%0A>%20>%20uniq.%20%20Like%20this:%0A>%20>%0A>%20>%20uniq%20-c%20-d%20-w%20list.of.files%0A>%20>%0A>%20>%0A>%20>%20This%20will%20print%20the%20list%20of%20files%20which%20are%20duplicates.%20%20For%20example,%0A>%20>%20out%20of%20a%20list%20of%2042,279%20files%20in%20a%20certain%20directory%20on%20my%20computer,%0A>%20>%20here's%20the%20result:%0A>%20>%0A>%20>%0A>%20>%0A>%20>%0A>%20>%0A>%20>%20_files/from_ebay_pc/pics_and_such_from_work/phone_backup/try2_nonptp_or_whatever/Pictures/Screenshots/Screenshot_20160312-113340.png%0A>%20>%0A>%20>%20Originally%20the%20_files%20directory%20had%20MANY%20duplicates,%20now%20I've%20managed%20to%0A>%20>%20get%20that%20down%20to%20the%20above%20list...%0A>%20>%0A>%20>%20Anyway,%20there%20you%20go.%20%20Happy%20scripting.%0A>%20>%0A>%20>%20---------------------------------------------------%0A>%20>%20PLUG-discuss%20mailing%20list:%20PLUG-discuss@lists.phxlinux.org%0A>%20>%20To%20subscribe,%20unsubscribe,%20or%20to%20change%20your%20mail%20settings:%0A>%20>%20https://lists.phxlinux.org/mailman/listinfo/plug-discuss%0A>%20>%0A>%20%0A>%20%0A>%20--%20%0A>%20:-)~MIKE~(-:%0A>%20---------------------------------------------------%0A>%20PLUG-discuss%20mailing%20list:%20PLUG-discuss@lists.phxlinux.org%0A>%20To%20subscribe,%20unsubscribe,%20or%20to%20change%20your%20mail%20settings:%0A>%20https://lists.phxlinux.org/mailman/listinfo/plug-discuss%0A>%20 "Reply to this message")
(text/plain)