Wednesday, March 2, 2011

Cleaning up your iTunes library (on a mac)

After merging two iTunes libraries I was left with many duplicates which iTunes asked me to delete one by one. I am always looking for efficient ways to solve boring tasks, so I went off to find a way to solve this automatically. Having a mac is great, because it has so many useful and easy tools to handle those kinds of problems. First of all, there is applescript, which I never touched before, but it really is the easiest way to deal with applications on a mac. To give you one example of its great powers consider the simple command you can enter in any applescript editor window

            tell application "iTunes" to play

Click "Run" and it plays! Easy, right?

Anyway, I wrote a few applescripts to clean up my library. I couldn't find similar scripts or at least not of the kind I wanted them to be (also, deleting duplicates is often available only for cash). I also wanted to learn applescript. So why not write my own scripts?

I want to share my experiences with other users and I hope they can be useful for more people out there. I trust on google to spread the word :) The scripts are (to some extent) optimized to be fast, using some tricks (or flaws) of applescript. Still, deleting the duplicates takes some time, especially for large libraries. This is however because of iTunes. Deleting is not a simple task for iTunes. It has to update quite a few things in the background, so dont be alarmed if you have to wait for some time. For me, with about 13.000 tracks deleting 6.000 duplicates took about one hour.

All scripts support (at least a simple) logging, i.e. you can check their progress by opening a log file, containing stuff like "Handling file xyz, Deleting file uvw". You can use it to check what is happening. You can easily change it and add more logging messages. You can also turn it off (by writing "false" after the doLogging property). Especially for the script "Clean up your media folder", you might want to try it with a smaller library first, to convince yourself that everything works finely. After all, your music files are valuable and you don't want them to be gone forever accidentally! However, my scripts don't really delete files, they just move them to the trash. So you can always save files, by checking there after the whole process.

Finally, here are some of the scripts:

  1. Delete duplicates. This little script finds and deletes duplicates in your iTunes library (It deletes the library entry, NOT the physical file. To delete the physical file, look at the "Clean up your music folder" script below. This is to avoid deleting files accidentally which you just added to your iTunes library, without copying to your music folder). Two tracks are considered duplicates if they share the same track name, artist and album title. Only if those 3 fields are the same, all of the tracks except one can be deleted. Additionally the play counts and the grouping entries are "merged" together, i.e. the highest play count among the duplicates is adopted and any grouping entry.  If you have better suggestions on how to merge tracks, I am open to any suggestions.
  2. Delete dead tracks. This script finds and deletes dead tracks in your iTunes library, i.e. tracks whose path do not point to an existing file in the file system.
  3. Clean up your music folder. This script finds and deletes files in your media folder which are not represented by entries in your iTunes library. I say "files" because you might have copied other files like pictures there and forgot about them or because maybe some other music program added hidden index files. If one of the folders in the media folder is empty, it is deleted as well.
    WARNING: You need to adapt the "libPath" to change the location of the media folder. By default it is set to "yourHomeFolder/Music/iTunes/Music". If you don't, ALL files in the specified folder will be deleted, because none of the files are present in YOUR iTunes library. You also might want to change the valid file endings, i.e. the list of file endings, which are considered to contain media data. ALL other files will be deleted for sure!!! Anyway, on a mac nothing is deleted immediately. All "deleted" files will be moved to the trash, so you can still save them if they were deleted incorrectly. Whatever you do, you should first check on a small library if the script does what you want. Don't blame me later, if all your data are gone!
Any suggestions are welcome. Hope my scripts help you learning apple script or handle your iTunes problem. And here finally the scripts. To run them, just copy and paste them to a new text edit window, save as "..".scpt. After clicking the apple script editor opens and you can click run. Alternatively you can save the files in the apple script editor as application, which then can be run by clicking like a regular app.

1. Delete duplicates


(*
This little script finds and deletes duplicates in your iTunes library. Two tracks are considered duplicates if they share the same track name, artist and album title. Only if those 3 fields are the same, all of the tracks except one can be deleted. Additionally the play counts and the grouping entries are "merged" together.
You can also use the main handler to delete duplicates in any other playlist than the main library.

You can turn on or off logging of deleted tracks. The logging goes to the "logFile.txt" in your home folder.

Any comments to
randolf.altmeyer@gmail.com
*)

property doLogging : false
property textlog : missing value

on run {}
tell application "iTunes" to set plList to library playlist 1
main(plList)
end run

on main(plList)
-- set up logger
set textlog to makeFileLog(((path to home folder) as string) & "logFile.txt")
if doLogging then textlog's logImportant("Starting deleting duplicates...")
findDuplicates(plList)
if doLogging then textlog's logImportant("Finished deleting duplicates...")
end main

on findDuplicates(plList)
tell application "iTunes"
-- find all artists
set artistList to artist of every track of plList whose artist is not missing value
set actArtList to {}
repeat with a in artistList
if not (actArtList contains (a as string)) then copy (a as string) to end of actArtList
end repeat
-- find all albums for a given artist
repeat with a in actArtList
set albList to (album of every track of plList whose artist is a and album is not missing value)
set actAlbList to {}
repeat with al in albList
if not (actAlbList contains (al as string)) then copy (al as string) to end of actAlbList
end repeat
-- find all tracks with artist a and album al
repeat with al in actAlbList
set trList to (name of every track of plList whose artist is a and album is al and name is not missing value)
set actTrList to {}
repeat with tr in trList
if not (actTrList contains (tr as string)) then copy (tr as string) to end of actTrList
end repeat
-- handle those tracks
repeat with tr in actTrList
set tracksWithThisComb to (every track of plList whose artist is a and album is al and name is tr)
if length of tracksWithThisComb > 1 then
my mergeTracks(tracksWithThisComb)
repeat with i from 2 to length of tracksWithThisComb
set t to item i of tracksWithThisComb
if doLogging then
set str to (tr & " - " & al & " - " & a)
textlog's logMessage("Deleting " & str)
end if
delete t
end repeat
end if
end repeat
end repeat
end repeat
end tell
end findDuplicates


on mergeTracks(possDupl)
tell application "iTunes"
set mainTrack to item 1 of possDupl
set playCount to played count of mainTrack
set grOfTrack to grouping of mainTrack
set gr to ""
set compOfTrack to compilation of mainTrack
set comp to ""
-- collect all information in mainTrack
repeat with i from 2 to length of possDupl
set tr to item i of possDupl
set trPlCount to played count of tr
if trPlCount > playCount then set playCount to trPlCount
if (grOfTrack is equal to "") then
set trGroup to grouping of tr
if not (trGroup is equal to "") then set gr to trGroup
end if
if (compOfTrack is equal to "") then
set trComp to compilation of tr
if not (trComp is equal to "") then set comp to trComp
end if
end repeat
set played count of mainTrack to playCount
if (grOfTrack is equal to "") then set grouping of mainTrack to gr
if (compOfTrack is equal to "") then set compilation of mainTrack to comp
end tell
end mergeTracks


on makeFileLog(file_path)
script FileLog
property class : "file log"
property _linefeed : character id 10
-- Writes log messages to a UTF8-encoded text file
on logMessage(the_text)
set f to open for access file_path with write permission
try
write (the_text & my _linefeed) to f starting at eof as «class utf8»
on error error_message number error_number
close access f
error error_message number error_number
end try
close access f
end logMessage
on logImportant(the_text)
logMessage("****** " & the_text & " ******")
end logImportant
end script
end makeFileLog



2. Remove dead tracks


(*
This script finds and deletes dead tracks in your iTunes library, i.e. tracks whose path does not point to an existing file in the file system.

You can turn on or off logging of deleted tracks. The logging goes to the "logFile.txt" in your home folder.

Any comments to
randolf.altmeyer@gmail.com
*)

property doLogging : false
property textlog : missing value

on run {}
main()
end run

on main()
-- set up logger
set textlog to makeFileLog(((path to home folder) as string) & "logFile.txt")
if doLogging then textlog's logImportant("Starting deleting dead tracks...")
tell application "iTunes"
set locations to location of every track of library playlist 1
set allTracks to every track of library playlist 1
set allNames to name of every track of library playlist 1
repeat with i from 1 to length of locations
set loc to item i of locations
if (loc is equal to missing value) then
set n to item i of allNames
if doLogging then textlog's logMessage("Deleting dead track " & n)
delete item i of allTracks
end if
end repeat
end tell
if doLogging then textlog's logImportant("Finished deleting dead tracks...")
end main

on makeFileLog(file_path)
script FileLog
property class : "file log"
property _linefeed : character id 10
-- Writes log messages to a UTF8-encoded text file
on logMessage(the_text)
set f to open for access file_path with write permission
try
write (the_text & my _linefeed) to f starting at eof as «class utf8»
on error error_message number error_number
close access f
error error_message number error_number
end try
close access f
end logMessage
on logImportant(the_text)
logMessage("****** " & the_text & " ******")
end logImportant
end script
end makeFileLog



3. Clean up music folder


(*
This script finds and deletes files in your media folder which are not represented by entries in your iTunes library. I say "files" because you might have copied pictures there and forgot or because maybe some other music program added hidden index files. If one of the folders in the media folder is empty, it is deleted as well.

You can adapt the "libPath" to change the location of the media folder. By default it is set to "yourHomeFolder/Music/iTunes/Music". You also might want to change the valid file endings, i.e. the list of file endings, which are considered to contain media data. ALL other files will be deleted for sure!!!
Anyway, on a mac nothing is deleted immediately. All "deleted" files will be moved to the trash, so you can still save them if they were deleted incorrectly.
Whatever you do, you should first see on a small library if the script does what you want. Dont blame me later, if all your data are gone!

You can turn on or off logging. The logging goes to the "logFile.txt" in your home folder.

The associative list code is from the excellent book "Learn apple script" by Hanaan Rosenthal.

Any comments to
randolf.altmeyer@gmail.com
*)

property doLogging : false
property textlog : missing value
property libPath : ((path to home folder) as string) & "Music:iTunes:Music"
property validFileEndings : {".mp3", ".wav", ".m4p", ".m4a", ".ogg", ".avi"}


on run {}
main()
end run

on main()
-- set up logger
set textlog to makeFileLog(((path to home folder) as string) & "logFile.txt")
try
cleanupMediaFolder(libPath)
on error m
textlog's logError(m)
end try
end main

on cleanupMediaFolder(libPath)
if doLogging then textlog's logMessage("Starting cleaning up media folder...")
tell application "iTunes"
-- list all files in the library
set trackPaths to location of every track in library playlist 1
-- put them in an associative list
set lib to my make_associative_list()
repeat with trPath in trackPaths
my addFileToList(trPath, lib)
end repeat
-- check for every file in the media folder if its path is contained in the list above
set allFilesInMediaFolder to my listFilesRecursively(libPath)
set AppleScript's text item delimiters to ":"
if doLogging then textlog's logMessage("Checking files in media folder...")
repeat with fPath in allFilesInMediaFolder
if doLogging then textlog's logMessage("Handling " & fPath)
-- check if every single path element can be found in lib
set pathItems to text items of fPath
set found to true
set curLst to lib
repeat with i from 1 to (length of pathItems)
set pathItem to item i of pathItems
if (i is equal to (length of pathItems)) then
if not (curLst contains (pathItem)) then
set found to false
end if
else
set lst4pathItem to getItem(pathItem) of curLst
if (lst4pathItem is equal to missing value) then
set found to false
exit repeat
end if
set curLst to lst4pathItem
end if
end repeat
-- every file which is not contained, can be deleted
if not found then
if doLogging then textlog's logMessage("Deleting " & fPath)
tell application "Finder" to delete file fPath
end if
end repeat
if doLogging then textlog's logMessage("Finished checking files in media folder.")
set AppleScript's text item delimiters to {""}
-- delete empty folders
my deleteFoldersRecursively(libPath)
end tell
if doLogging then textlog's logMessage("Finished cleaning up media folder...")
end cleanupMediaFolder

on deleteFoldersRecursively(pathToFolder)
set fileList to {}
tell application "System Events"
-- handle files
set filesList to path of every file in folder pathToFolder
repeat with pathOfF in filesList
set foundValidFileEnding to false
repeat with fileEnding in validFileEndings
if (pathOfF ends with fileEnding) then
set foundValidFileEnding to true
exit repeat
end if
end repeat
if foundValidFileEnding then copy pathOfF to end of fileList
end repeat
-- handle folders
set foldersList to path of every folder in folder pathToFolder
repeat with pathOfF in foldersList
set fileList to fileList & my deleteFoldersRecursively(pathOfF)
end repeat
if fileList is equal to {} then
if doLogging then textlog's logMessage("Deleting folder " & pathToFolder)
delete folder pathToFolder
end if
return fileList
end tell
end deleteFoldersRecursively

on listFilesRecursively(pathToFolder)
set fileList to {}
tell application "System Events"
-- handle files
set filesList to path of every file in folder pathToFolder
repeat with pathOfF in filesList
set foundValidFileEnding to false
repeat with fileEnding in validFileEndings
if (pathOfF ends with fileEnding) then
set foundValidFileEnding to true
exit repeat
end if
end repeat
if foundValidFileEnding then copy pathOfF to end of fileList
end repeat
-- handle folders
set foldersList to path of every folder in folder pathToFolder
repeat with pathOfF in foldersList
set fileList to fileList & my listFilesRecursively(pathOfF)
end repeat
return fileList
end tell
end listFilesRecursively


on addFileToList(f, lst)
set fPath to (f as string)
set AppleScript's text item delimiters to ":"
set pathItems to text items of fPath
set AppleScript's text item delimiters to {""}
set curLst to lst
repeat with i from 1 to ((length of pathItems) - 1)
set pathItem to item i of pathItems
set lst4pathItem to getItem(pathItem) of curLst
if (lst4pathItem is equal to missing value) then
if (i is equal to ((length of pathItems) - 1)) then
set lst4pathItem to {}
else
set lst4pathItem to make_associative_list()
end if
curLst's setItem(pathItem, lst4pathItem)
end if
set curLst to lst4pathItem
end repeat
copy (last item of pathItems) to end of curLst
end addFileToList

on makeFileLog(file_path)
script FileLog
property class : "file log"
property _linefeed : character id 10
-- Writes log messages to a UTF8-encoded text file
on logMessage(the_text)
set f to open for access file_path with write permission
try
write (the_text & my _linefeed) to f starting at eof as «class utf8»
on error error_message number error_number
close access f
error error_message number error_number
end try
close access f
end logMessage
on logImportant(the_text)
logMessage("****** " & the_text & " ******")
end logImportant
on logError(the_text)
logMessage("ERROR: " & the_text)
end logError
end script
end makeFileLog

on make_associative_list()
script AssociativeList
property class : "associative list"
property the_items : {}
property theKeys : {}
property theKeysRef : a reference to theKeys
on getKeys()
set theKeys to {}
considering diacriticals, hyphens, punctuation and white space but ignoring case
repeat with record_ref in my the_items
set keyOfRec to key of record_ref
copy keyOfRec to end of theKeysRef
end repeat
end considering
return theKeys
end getKeys
on find_record_for_key(the_key)
(* This is a private handler. Users should not use it directly. *)
considering diacriticals, hyphens, punctuation and white space but ignoring case
repeat with record_ref in my the_items
if key of record_ref = the_key then return record_ref
end repeat
end considering
return missing value
end find_record_for_key
on setItem(the_key, the_value)
set record_ref to find_record_for_key(the_key)
if record_ref = missing value then
set end of my the_items to {key:the_key, value:the_value}
else
set the_value of record_ref to the_value
end if
return
end setItem
on getItem(the_key)
set record_ref to find_record_for_key(the_key)
if record_ref = missing value then
--error "The key wasn't found." number -1728 from the_key
return missing value
end if
return value of record_ref
end getItem
on count_items()
return count my the_items
end count_items
on delete_item(the_key)
set record_ref to find_record_for_key(the_key)
if record_ref is missing value then
error "The key wasn't found." number -1728 from the_key
end if
set contents of record_ref to missing value
set my the_items to every record of my the_items
return
end delete_item
end script
end make_associative_list

5 comments:

  1. had a little bug in the "delete duplicates" script...I wrote "log_message" instead of "logMessage". Should be fine now.

    ReplyDelete
  2. hi, I got this response for the remove duplicate:

    error "iTunes got an error: File permission error." number -54

    it highlighted this line:


    set played count of mainTrack to playCount


    any idea why? it can be very useful if it would work...

    Thanks.

    ReplyDelete
  3. i liked your free scripts. many thanks for providing!

    ReplyDelete
  4. Thanks for your tutorial, but it's something complex. For many novice, I prefer to use Leawo Tunes Cleaner to clean up iTunes library and delete duplicates. It's easy to use and can complete music tags with simple clicks.

    ReplyDelete