Scan folders/file scanner confusion

[patch]

From 0.9.9.x Beta --> 0.9.9.3 New plugin system

Why would I spend time manually preparing files in such a way that I wouldn't have to use the built-in automatic functions? It is also on the edge of the scope of a file scanner to retrieve info not from the file itself but from an array of files paired with the video file.

I do not believe anyone intends to manually generate such files.
You are correct PVD current internet search capability is the best way of collecting such information.
But there use is the same as id3 tags for music files.
As such they are best written by an export command from within PVD.
They could then be read by any other instance of PVD (eg another user on the same computer or same network scanning a shared directory, or movies copied to a completely separate network). Once movie descriptions are stored near movie files they can also be used to transfer information to other programs (the same as is currently done with music files). It is correct that writing tag information to internal tags would be preferable, but that is not supported by mediainfo.sourceforge.net and unlikely to be supported within the foreseeable future see http://sourceforge.net/forum/forum.php?thread_id=1695277&forum_id=297609

[blue334]

I think what you're asking can largely be acheived using the current export plugin.
What you're describing is essentially an advanced html export template.

Where you lose me is where you suggest this kind of information storage should be read by PVD directly. PVD is after all a database app, you cannot acheive anything like the current functionality outside the database engine. A browser would be better suited.

[patch]

I think what you're asking can largely be acheived using the current export plugin.
What you're describing is essentially an advanced html export template.

When I last tried (0.9.8.20), the export function went close. It was able to create 1 file per media entry with useful data in it. However what I could not achieve is to put this file near the media file (same directory & name as media file but different file extension) as the path variable wanted an absolute address. I didn't try exporting poster images the same location but with a jpg extension, but believe that to would be useful.

So yes I believe I am after an enhancement to the export function.

Where you lose me is where you suggest this kind of information storage should be read by PVD directly. PVD is after all a database app, you cannot acheive anything like the current functionality outside the database engine. A browser would be better suited.

It's value is in transferring information between instances of PVD or to another similar application.
The ways I intend to use it is
1) I store my multimedia an shared computer drives. When I add movie to the collection I scan it into PVD, check the display name, link to imdb, and cover are correct.
2) I would like to be able to write this information back to the movie (or it's directory)
3) There are many other media users in this household who access the movies from the multiple user accounts on the multiple computers. To avoid having them redo step 1, PVD could just scan and read the data written in step 2 (it already has the ability to scan a directory on start up, and read information from a text file near the movie file).
4) Similarly if some of the movies were copied to another server, where different users ran PVD, then they too could scan the identifying data into their copy of PVD automatically.
5) I believe media centre is also likely to be able to read tags of this sort, enabling efficient transfer of information between applications

This is the same functionality achieved by tagging mp3 files, and basically has the same advantages.

[nostra]

The export plugin will be improved with an extra option to save files near video files in the next version.
I am also planning an export template and some additional import functionality (including file scanner) using predefined XML files near video files.

[patch]

Tag export/import (like ID3 for mp3) has been marked as already added. This is good news if that is the case. Looking at 0.9.9.10 I can't see where it is implemented.
Further to discussions here

Actually I'm not sure what is going to be the most efficient path to getting movie tagging functionality.
Importing & exporting, text & image data near a movie file is part of the functionality required.
However I suspect working reliably with the file scanners name parsing and multi-part file identification could well be more challenging.

To explain
For a single file movie the tag information would probably go in .jpeg and .txt (or .xml or .pvd?) files with identical path and base file name.
For multi-file movies the base file name is less obvious as it would involve stripping out the disc number or using the first file name or exporting for all component files
For DVD images (ie with .vob files) the tag files could go within the directory containing the .vob files or 1 level above; with a generic file name or directory name (if 1 level above .vob files)
For TV series similar issues arise however then there is the need to identify the episode, season, series, as well as grouping with other episodes in the same series.

As such it wouldn't surprise me if nostra intentionally left it till other parts of the file scanner were implemented & stable.

mmm
Reading through this again, it looks as if I'm making it all too hard so perhaps it is time to stand back & think what it is all about.
My aim is for PVD to import media with maximum reliability and minimum user intervention.
The problem is PVD needs to make a connection between the video data I have & various internet databases but PVD can't watch movies and it is often ambiguous which record in and internet database is preferred. So an indirect link needs to be used. The best way of doing this is going to vary with the information available to PVD, for example:

1) Analyse file (& path) name to deduce movie name, then use that to look up internet databases. PVD does this very well (& flexibly with regex). The only enhancement I can think of is sometimes the enclosing directory has a better name, so being able to easily select that would be handy.

2) Some media files have an associated .nfo file containing a link to an internet database (most often imdb). It would be good if PVD could automatically extract the reference. I'm actually not sure if it is doing this already as it has been discussed. Knowing what .nfo file to associate with what media file could be a problem as they are generated by humans, thus done inconsistently. The way to start is support the easy cases ie if have .nfo with same base file name as media or folder with one .nfo & one media file then PVD could assume they belong together. Ideally PVD would display some how that it found the reference in the "Scan results" dialogue box. Perhaps the action could show "Add movie" imdb link. This would save the user worrying about the information in the file name.

3) If the movie has already been imported into PVD, and links to internet databases checked, then it would be good if that information could be written back to the movie file, so if it was moved to another system, it could then be imported into PVD there with minimal user intervention. Writing data back to the movie file is also likely to assist communication between media programs (is it does for music with mp3 tags). For PVD though the value is to provide assistance to the file scanner, so the writing protocol needs to be linked to the file scanner reading protocol. Again just doing the easiest case (single movie file) would be useful (provided it cleanly ignored the more complicated case and its functionality was documented eg in change log)

[rick.ca]

Tag export/import (like ID3 for mp3) has been marked as already added. This is good news if that is the case. Looking at 0.9.9.10 I can't see where it is implemented.

I marked that off because I did not interpret the feature request to necessarily be an integrated tag management system, and I thought most of the pieces were in place for this to be done on an ad hoc basis. Perhaps I'm wrong, but it doesn't seem you've actually tried to do anything with the tools that have been provided. I'm not trying to give you a hard time, but since I don't really have much interest in this, I was expecting you would give it a shot.

Is there not an template command for exporting text file beside a media file? That could be used to export a text file with the URL. I assume the Text File plugin can import that. Those two could export/import any text, but, as I understand it, the URL is your primary concern—with that, another instance of PVD would not have to search IMDb to find the movie. I believe there's also a template command for exporting posters beside media files. And there's Image by ID to reassociate them with a new database. That would work only for new databases, so I do understand your interest in Reset's plugin.

The only enhancement I can think of is sometimes the enclosing directory has a better name, so being able to easily select that would be handy.

You should be able to handle that with regex as well, unless the pattern is hopelessly inconsistent. What is the source of the directory and file name?

Some media files have an associated .nfo file containing a link to an internet database (most often imdb).

I don't see much point in this. Sure, it there's a reliable URL consistently available in a consistent format, then this may help, but the primary approach is supposed to be to get the filename (and maybe year) and use that to search IMDb. In normal circumstances, that works fairly well. What if the URL is wrong? In that case, a mess would have to be undone, and the normal method repeated—after removing the offending NFO. Maybe what would be more practical would be to import the entire contents of any NFO into a custom field. Then, if any of the information was needed, it would be readily accessible.

[patch]

The only enhancement I can think of is sometimes the enclosing directory has a better name, so being able to easily select that would be handy.

You should be able to handle that with regex as well, unless the pattern is hopelessly inconsistent. What is the source of the directory and file name?

Media file names are often very cryptic eg "vmt-agmihtf-xvid.avi" but the enclosing directory is more informative "A Good Man Is Hard to Find". Maybe others don't have this experience, but I find it is rather common. Currently I cut & paste in the "Scan results" dialogue to make the name more informative for the imdb lookup. This is a wish list suggestion to improve to the "Scan results" dialogue to make this process more efficient. Regex would not make the manual editing more efficient as the very reason it is a manual process is the naming is generated by humans so is inconsistent.

Some media files have an associated .nfo file containing a link to an internet database (most often imdb).

I don't see much point in this. Sure, it there's a reliable URL consistently available in a consistent format, then this may help, but the primary approach is supposed to be to get the filename (and maybe year) and use that to search IMDb.

URL are in a consistent format (they are a valid url). If one is available it will always point to the correct imdb record with greater accuracy than a search (duplicate records in imdb, similar movie titles etc). So if it is available it is worth using, if not then falling back to file naming & database search is of course required. Currently I can "Open enclosing folder" open the .nfo file, cut & paste from there. Importing into a custom field would provide similar functionality but not address why I suggested it ie the improve scanner performance.

Note I have spent time making file names look pretty in the past, but given I now access my movies from PVD the actual file names and paths are largely irrelevant. If I need to change them to import into PVD then that indicates the efficiency of operating PVD could be improved.

.nfo are supplied with at least 50% downloaded content and most have imdb link, so I suspect I'm not the only one who would benefit from this functionality, hence it's suggestion as a scanner enhancement wish list item

Tag export/import (like ID3 for mp3) has been marked as already added. This is good news if that is the case. Looking at 0.9.9.10 I can't see where it is implemented.

I marked that off because I did not interpret the feature request to necessarily be an integrated tag management system, and I thought most of the pieces were in place for this to be done on an ad hoc basis.

PVD is run on an open database with various import & export capabilities. It has always been possible to import movie information from various sources in an adhoc fashion. The aim of using tags is to further improve the file scanner performance. To do so it becomes part of the file scanner. Maybe PVD has all the tagging capability nostra wanted to implement, if so then a some what adhoc solution must be the answer.

BTW
The above suggestion are in-order of solution specificity, from most general (but least exact) to most exact / potentially automatic.

[rick.ca]

Note I have spent time making file names look pretty in the past, but given I now access my movies from PVD the actual file names and paths are largely irrelevant. If I need to change them to import into PVD then that indicates the efficiency of operating PVD could be improved.

Obviously, the actual file names and paths are relevant. A lot of work went into a sophisticated file scanner that can accurately extract a title and year—which in turn will very likely result in a successful IMDb search. All that is required it that there be reasonable degree of consistency in the file path names. The fact that you're unwilling to use it as designed doesn't mean it's efficiency needs to be improved.

But fine. You're right, there are probably others who expected PVD to accurately process inconsistent, meaningless and misleading file path names used by many torrents. Most of the torrents that provide reliable NFOs are also more likely to use reasonable names, but let me be generous: Let's say that 50% of these torrent file names are so bad you really need a URL from an NFO. But only 50% of those have NFOs. So that means the you're still going to get an unacceptably high error rate.

I use "clean" file names, and the file scanner performs flawlessly. My file manager includes a regex file renamer. One button cleans movies, another episodes. Maybe I'll configure a third to deal with those that have a more accurate title in the containing folder. Even with some requiring manual adjustment, it still works out to seconds per file. And then the filenames are accurate and meaningful in my file system as well as in PVD. The same thing can be done with a number of excellent freeware file renamers.

The aim of using tags is to further improve the file scanner performance.

Maybe now that the scanner works so well with clean filenames, we don't need tags. Maybe what would be much more useful would be the ability to rename files. Files could be renamed to Title (Year) IMDb#.ext and a variable added so regex could capture the IMDb# and add the URL.

[patch]

My file manager includes a regex file renamer. One button cleans movies, another episodes. Maybe I'll configure a third to deal with those that have a more accurate title in the containing folder. Even with some requiring manual adjustment, it still works out to seconds per file. And then the filenames are accurate and meaningful in my file system as well as in PVD. The same thing can be done with a number of excellent freeware file renamers.

That sound like a good system.
I got sick of doing it by hand. The issue I found was often multi-part files were used and for replay all files names needed to match (sub-titles, multiple discs etc). As such it was easier to leave the cryptic but matching set of file names. Typically the folder names are more human readable, hence my approach.

Does the file manager you use cope with multi-part files appropriately? Which one do you use?
You do of course realise there is a certain amount of similarity between what you are doing with your file manager and what I (& I assume others) are trying to do with PVD file scanner.

But only 50% of those have NFOs

imo PVD being able to link directly to the exact imdb reference would be a very useful feature. Currently I do this by hand which is much less efficient.
True if I knew the Exact imdb title & year the text search would work well, but often I do not know this prior to looking it up on imdb.

Maybe what would be much more useful would be the ability to rename files. Files could be renamed to Title (Year) IMDb#.ext and a variable added so regex could capture the IMDb# and add the URL.

That is a neat idea.
We would need to ensure it worked with multi-part files.
I would like at least the option to append the old file name ending to the new cleaned name, maybe Title (Year) IMDb# [old descriptors].ext as I'm not sure I would really like to loose some of that information (CAM, TS, dvdrip, HDTV, BluRay)
Not sure if it would cope with graphics data but given the often reasonable image PVD can pull from imdb I still think it is a good approach.

[rick.ca]

Does the file manager you use cope with multi-part files appropriately? Which one do you use?

Directory Opus. It's renamer only does what the regex I specify does (it also has other modes, like find & replace, and script—where file attributes and tags could be used). I haven't run into problems with multi-part files. When I do, maybe I'll be inspired to modify the regex.

You do of course realise there is a certain amount of similarity between what you are doing with your file manager and what I (& I assume others) are trying to do with PVD file scanner.

Yes, I thought about that and concluded that just because a similar function is required at two different stages of my workflow does it mean it has to be restricted to one or the other. It does mean that something has to be done at the first stage. You don't have to be very sophisticated at the file renaming stage—just change it so the essential variables (title, year, season, episode, disc#) are in a fixed pattern. Then the second stage regex can't miss—they're just pulling out those variables from a fixed pattern. Also, I should emphasize the renaming stage is not automatic, but under my supervision. So I'm also using my own judgment as to what is required. That may range from tweaking the regex and running it on 100 new files, to deciding the one file name I'm processing right now is easier to edit manually.

BTW, all this is explained in 0.9.9.x File Scanner and Regular Expressions. This post includes useful regex references and a link to a good freeware renamer.

True if I knew the Exact imdb title & year the text search would work well, but often I do not know this prior to looking it up on imdb.

Torrent file names often include a year (which is easy to identify with regex). It seems to me that even if the title is inexact, including the year dramatically improves the IMDb search result.

I would like at least the option to append the old file name ending to the new cleaned name, maybe Title (Year) IMDb# [old descriptors].ext as I'm not sure I would really like to loose some of that information (CAM, TS, dvdrip, HDTV, BluRay)

Just specify the regex that will do so. Instead of skipping over them (or, for example, dropping everything after finding the title and year), put those parts in variables and append them as [old descriptors].

Not sure if it would cope with graphics data but given the often reasonable image PVD can pull from imdb I still think it is a good approach.

Don't forget it's now very easy to export posters to images "beside" the media file. So if a lot of effort has gone into getting better quality posters from elsewhere, this is a reasonable thing to do—to make those available to other PVD databases and/or just to be a visual reference in the file system.

A nice companion to a file renaming feature would be the option to save the poster beside the file at the same time.

Nostra, I'm warming up to the idea of starting a 1.0 wish list. It seems I now have a number of feature requests that could go with my recommendation for Scanners (1981).

[patch]

You do of course realise there is a certain amount of similarity between what you are doing with your file manager and what I (& I assume others) are trying to do with PVD file scanner.

Yes, I thought about that and concluded that just because a similar function is required at two different stages of my workflow does it mean it has to be restricted to one or the other.

mmm
PVD already has regex, it already stores user defined regex expressions for the file scanner.
As a wish list item I would like to combine the essential features of rick.ca file renamer into PVD. To explain:-
The "Scan results" dialogue should show the fields it will potentially use to search the online databases & set up the record. For movies these are
Title
Original title
Year
url
Path

To that I would like added the ability to run user defined regex (via buttons or menu selection). Typically they would take the "path" as the input string & output to the above fields as appropriate. To be general it maybe worth having tha ability to start from other fields. I imagine some may prefer to run a regex on the existing Title to clean it.
It would also be useful to make the url active so the linked page could be viewed in the users default browser.

For series a similar thing could be done, but more fields are required
Season
Episode
Episode title

The user needs also to be able to correct the scanner if it gets it wrong. A possible solution is to have a series check box, which when ticked indicates the video is part of a series and the extra fields are shown (Unless it is felt the scanner would always be correct). I have not thought through how to incorporate the add to series functionality, but that too would be applicable here.

The reason I would prefer it as part of PVD is I like the way PVD file scanner just shows me the new or changed files when I point it a at directory trees. I also think a dedicated tool like PVD could do a better job of managing video files.

The actual file name updating could then be done as a second step, useful for those who look at the file names or want to ensure subsequent accurate re-scanning.

I would like at least the option to append the old file name ending to the new cleaned name, maybe Title (Year) IMDb# [old descriptors].ext as I'm not sure I would really like to loose some of that information (CAM, TS, dvdrip, HDTV, BluRay)

Just specify the regex that will do so. Instead of skipping over them (or, for example, dropping everything after finding the title and year), put those parts in variables and append them as [old descriptors].

Sorry I didn't explain myself, I was thinking how I would like a PVD file renamer to work, not how Directory Opus works

Not sure if it would cope with graphics data but given the often reasonable image PVD can pull from imdb I still think it is a good approach.

Don't forget it's now very easy to export posters to images "beside" the media file. So if a lot of effort has gone into getting better quality posters from elsewhere, this is a reasonable thing to do—to make those available to other PVD databases and/or just to be a visual reference in the file system.

A nice companion to a file renaming feature would be the option to save the poster beside the file at the same time.

Good ideas

[rick.ca]

The reason I would prefer it as part of PVD is I like the way PVD file scanner just shows me the new or changed files when I point it a at directory trees. I also think a dedicated tool like PVD could do a better job of managing video files.

I don't think there's any advantage in trying to turn PVD into a file manager. Don't be mislead by the fact I use regex in my file manager. Those regex can do no more or less than those in the scanner configuration. The difference is, if there anything that needs to be done manually, the file manager is the appropriate environment in which to do it. If I had a good eye and didn't care about the file names, I could just make the bare minimum of changes necessary so the scanner regex would work. As a practical matter, renaming the files in the file manager not only "cleans" them, but makes it obvious which ones need manual adjustment so that the scanner regex will work.

The function of the scan results dialog should be restricted to correcting associations for movies already in the database, or enabling a more accurate search. In other words, it's for moving forward to the next step, not for fixing errors from the previous step. In a well designed work flow, this dialog should more often than not just be a simple confirmation.

Sorry I didn't explain myself, I was thinking how I would like a PVD file renamer to work, not how Directory Opus works

A PVD renamer would be rename files based on user-configurable database variables like Title, Year, IMDb#, etc. To use such "other information," an additional regex variable would have to be provided that you could use to save the data in a custom field. Then, of course, the renamer would have to allow variables for custom fields. But allowing any custom field to be used in a file name doesn't seem like a good idea. Maybe the Features field could be used for this purpose.

[nostra]

OK, I will be laconic:

1. I do not think there will be some built-in support for tagging. I only think of realizing this functionality with scripts/plugins

2. There will be file rename feature, but only to organize files for existing movies in database

3. I am very curious about your 1.0 feature suggestions, Rick

[rick.ca]

3. I am very curious about your 1.0 feature suggestions, Rick

I haven't thought about it much, other than in this conversation. But to summarize:

1. The scan results dialog should allow the user to make changes...

a) For records already in the database, forcing the match in cases where the scanner "missed" and thinks a file is new. This would include forcing a match of an episode to an existing series when the scanner has failed to recognize it as an episode.

b) For records not in the database, changes to title and year, to increase the likelihood of a successful web search.

2. As noted here, the scanner should not identify files as series unless a season and episode is determined by the regex.

3. The introduction of a regex variable for storing an online database code from which a URL can be constructed from base URL mask saved in the user's configuration. For example, an IMDb mask could be http://www.imdb.com/title/tt[URL code]/, where [URL code] is the 7-digit IMDb title code.

4. A file path renaming utility which uses relevant database fields as variables. This sort of thing can be rather dangerous unless the user can see the results before committing—it needs some kind of "old path ~ new path" preview, or perhaps some kind of integration with a reintroduced Gridview (where a preview can be shown in a column beside the existing file path). Examples of what should be possible:

[parent]\[title] ([year]) [rating] [url code].[ext]
[parent]\[title] ([year])\{DVD rip files}
[parent]\[series] S[s#] E[e#].[ext]
[parent]\[series]\Season [s#]\[e#] [episode title].[ext]

5. Since some (like patch) will use a renaming utility to ensure other users/databases will be able to get online information with 100% accuracy, it would be nice to also save the current poster using the same name—at the same time the renaming is done (i.e., as an option, "Save poster with same pathname?").

I think a file renamer should have the effect of making users much more accepting of the scanning process. They only need to change new filenames to get results that are "good enough" (that depends on the user—how good their regex are and how willing they are to make corrections in the results dialog). Once the "processing" is done, they will be able to use the renamer to make the file pathnames "perfect."

[Nebol]

In the case of episodes, there are so many different ways they may be organized and named, it's probably easier for all if you decided how it should be done. I like Dadeo's ..\Title\Title.S02E01.episodename.avi, especially if, with that format, your routine could consistently identify titles—whether the episode files are in a separate sub-folders or not.

Problem is, that would not work at all for me, since, in my system, I definitely can not have periods in the file names, only before the suffix. I use underscores. Title_S02E01_whatever.suffix
Also, using periods like that is damn ugly.

[nostra]

What system are you using that would not let you use periods in file names???

[rick.ca]

The periods are irrelevant. They could also be underscores, spaces or pipes. The point is there just needs to be a consistent structure to the filenames so the regex can work.

[patch]

The point is there just needs to be a consistent structure to the filenames so the regex can work.

We could probably do that by listing (default) supported formats in the wiki.
Currently this aspect is avoided
http://www.nimidia.com/pvd_wiki/tiki-index.php?page=Organize-files-by-episode
http://www.nimidia.com/pvd_wiki/tiki-index.php?page=Adding-a-New-Movie&structure=PVD-Manual

But I may have missed it as I'm having problems using the search function in the wiki.

[rick.ca]

We could probably do that by listing (default) supported formats in the wiki.
Currently this aspect is avoided

That would just encourage the misconception the program is intended to handle any filenaming convention one cares to use. I'd rather just say a consistent filenaming convention like Title (Year).ext for movies and SeriesTitle S00 E00 [EpisodeTitle].ext for series will be foolproof. The regex provided will handle many forms in addition to these. The regex can even handle a somewhat inconsistent filenaming convention, but that gets complicated very quickly. Considerable care has to be taken to ensure the regex do not conflict with one another, and are applied in the correct order.
 
In fact, a list of "supported formats" would be misleading unless it also enumerated all the possible conflicts. This recent post is a good example of the difficulty. You can name movie files Year - Title [(Director]).ext if you like, but then you have to provide a regex that will recognize this pattern (including the Director part, whether or not it exists) and remove (or reorder) any regex that would misinterpret the year as an episode number.

[patch]

We could probably do that by listing (default) supported formats in the wiki.
Currently this aspect is avoided

That would just encourage the misconception the program is intended to handle any file naming convention one cares to use.

Not if it was listed as

PVD is designed to support...
I can in fact extract the required information if you use other naming conventions but you may need to manually edit the title during "scan folders for new movies / changed file paths" or modify the regex expressions in "Tools -> preferences -> File scanner". See ... for further details

My intention is we would document the default / preferred naming. Illustrating some of the flexibility in punctuation & capitalisation, but not covering every possibility as that is best done by reading the regex search terms.
Sorry I didn't make that clear.