Scan folders/file scanner confusion

[Dadeo]

I admit I am new to the program, but I am at a loss how to use the Scan folder/file function properly.

In the case of TV Series, sometimes the scanner puts all the file paths for all the episode files into the file path field for the existing Series record instead of putting one file path in each episode record.  Is that what it is supposed to do?  Sometimes it puts them in separate records for each Episode but then it does not group them by series! The only thing I can see is that it depends on how the files are named. If the filenames do not have spaces, they seem to all go in the one record (and it takes forever!). If they have spaces, then they go in separate records, but never in episode records (this is much faster)!

In the case of Movies with more than one file, it tries to create a separate movie record for each file, instead of putting both files in the File Path. Again this may be because there are spaces in the filename.  I don't know!

Please help!

[nostra]

In the case of TV Series, sometimes the scanner puts all the file paths for all the episode files into the file path field for the existing Series record instead of putting one file path in each episode record.  Is that what it is supposed to do?  Sometimes it puts them in separate records for each Episode but then it does not group them by series! The only thing I can see is that it depends on how the files are named.

The file scanner does not fully support episodes right now, so it will put all the file to the main movie record in the most cases. The scanner has a complex algorithm for detecting where to put files and how to combine them, but it is certainly not perfect. I plan full episodes support in file scanner and some additional scan options for future versions.

How do you name your files? Could you post some examples that produce different results?

[Dadeo]

Usually I name my files as follows:

...\Title\Title.S02E01.episodename.avi

File scan puts all these in the File Path of the main Series record with a vertical bar | separating each file.

However, I forgot to set the Preferences to leave the Titles Formatted as they are and now I have to change them all back manually or else File Scan thinks it is a new series (most often where the series name starts with 'The') 

Some of the files are named:
\Title\Season\Title - 1x02 - Episodename - details.avi

File scan puts these into separate series records, not episodes, even if there is already a Series record with the Title name.

I hope that helps.  If you need any more detail let me know.

[rick.ca]

The scanner has a complex algorithm for detecting where to put files and how to combine them, but it is certainly not perfect...

Your programming skills never cease to amaze me, but I have a hard time seeing how this ever will be perfect. I think you have the "right" to dictate to users a file naming convention to be used for files to be recognized by the program. Hopefully it would be flexible, but surely you need some degree of consistency. And users need to know what will be recognized. Some will have large collections named in a manner that doesn't comply, but it's not very difficult to rename files using a batch renaming utility (or a "home-made" batch file).

In the case of episodes, there are so many different ways they may be organized and named, it's probably easier for all if you decided how it should be done. I like Dadeo's ..\Title\Title.S02E01.episodename.avi, especially if, with that format, your routine could consistently identify titles—whether the episode files are in a separate sub-folders or not.

[nostra]

Your programming skills never cease to amaze me, but I have a hard time seeing how this ever will be perfect. I think you have the "right" to dictate to users a file naming convention to be used for files to be recognized by the program. Hopefully it would be flexible, but surely you need some degree of consistency. And users need to know what will be recognized. Some will have large collections named in a manner that doesn't comply, but it's not very difficult to rename files using a batch renaming utility (or a "home-made" batch file).

You are right, it is impossible to include all naming conventions, but it is possible to enable users to setup how PVD should parse the file names.

In the case of episodes, there are so many different ways they may be organized and named, it's probably easier for all if you decided how it should be done. I like Dadeo's ..\Title\Title.S02E01.episodename.avi, especially if, with that format, your routine could consistently identify titles—whether the episode files are in a separate sub-folders or not.

This format will be certainly supported.

[rick.ca]

but it is possible to enable users to setup how PVD should parse the file names.

But then it would be my fault if it didn't work! 

[Dadeo]

It may not be possible to recogiize ALL filenames, but it is possible to recognize and parse most of them.  I use a plugin in MediaPortal (which is fully free and Open Source) called MyTVSeries which is excellent.  It automatically links to my AMC catalog in XML format. You should consider an XML format for PVD that could be used by this plugin because MediaPortal is getting more widely known and used for the excellent Media Center it is.  You should get PVD known by these people.  This plugin does use SQL for the database I believe, but I don't know what the actual code the plugin is written in.

When parsing files, I use a regexp in the following format and it parses almost every file I have no matter how it is named:

^.*?\\?(?<series>[^\\$]+?)(?:s(?<season>[0-1]?\d)e(?<episode>\d\d)|(?<season>(?:[0-1]\d|(?<!\d)\d))x?(?<episode>\d\d))(?!\d)(?:[ .-]?(?:s\k<season>e?(?<episode2>\d{2}(?!\d))|\k<season>x?(?<episode2>\d{2}(?!\d))|(?<episode2>\d\d(?!\d))|E(?<episode2>\d\d))|)[ -.]*(?<title>(?![^\\]*?sample)[^\\]*?[^\\]*?)\.(?<ext>[^.]*)$

I don't know if this helps PVD in any way, but I just thought I'd throw it out there in case!

I agree it would be helpful to know what filenames PVD can parse!

[Dadeo]

OK, I have another problem/question!

In an existing database, when I have movies with two files (usually named Title\Title.ext -cd1 and Title\Title.ext -cd2) the filescan wants to change the file path to the second filename.  (In some cases it wants to add a new movie!).  Is there any way I can tell it to append the file to the existing File Path field like it does when I run filescan on a new database (e.g. Title\Title.ext -cd1|Title\Title.ext -cd2)

If not, could this feature be added (i.e. option to append filename to file path) where it matches an existing title?

If there are two file names in the File Path, will PVD play both files (I am guessing PVD uses the default player, so this likely depends on the player!)

If not, I guess I just need to manually add all the second file names to the File Path field or put them in the Ignore List?

Don't get me wrong, I think the FileScan feature is FANTASTIC. Especially when it automatically checks for new movies and enters them in my database. That's why I just want it to work more seamlessly!

I am still trying to think of a solution where the Title parsed from the Filename cannot match the Title field (usually because it contains punctuation which cannot be included in a filename such as a colon).

[nostra]

In an existing database, when I have movies with two files (usually named Title\Title.ext -cd1 and Title\Title.ext -cd2) the filescan wants to change the file path to the second filename.  (In some cases it wants to add a new movie!).  Is there any way I can tell it to append the file to the existing File Path field like it does when I run filescan on a new database (e.g. Title\Title.ext -cd1|Title\Title.ext -cd2)

No, it is not possible


I will review all the suggestion on this topic and improve the File Scanner accordingly in the next major release. Please, be patient.

[Dadeo]

Thanks Nostra, don't worry I am just learning, not impatient!  I appreciate all your great and timely support and the efforts you are making!

What I did was create a PVD using file scan, then import all the results into an AMC database then import all the file paths from that AMC database into my PVD database.  I had to use the AMC import because I don't think the CSV import lets you merge or overwrite data on matching titles.  Am I right?  It would be great if the CSV import could do this as well 

Just a note on a few weird problems on the filescan parsing in Titles:

1.  10,000 becomes 10,00
2.  3 10 becomes 3 0
3.  Mr. (also Mrs.) becomes Mr (Mrs) looks like all dots (periods) are stripped

These files may or may not import movie info from the web depending on the import site.  If I correct the titles, filescan keeps finding them as new movies.

[nostra]

If there are two file names in the File Path, will PVD play both files (I am guessing PVD uses the default player, so this likely depends on the player!)

PVD will play both files using a temporary .pls file. It works with most video players.

[patch]

I admit I am new to the program, but I am at a loss how to use the Scan folder/file function properly.
...
In the case of Movies with more than one file

I to are confused, and would like to know how I should name files for the scanning to work in 3 different situations.

1) Movie over more than one disc. It seams to identify "Shrek cd1.avi" "Shrek cd2.avi" etc as multiple discs for the same movie.

2) Later movies in a series I usually put in the same directory (eg "Shrek 1.avi", "Shrek 2.avi", "Shrek 3.avi"). It seams to assume it is all one movie over multiple files so combines them. I'm not sure how I should name them to prevent this. Also is there an efficient way of splitting up the enteries after they have been scanned (and falsely grouped)?

3) Movies with a sample file / trailer eg "Shrek - sample.avi", "Shrek.avi" are assumed to be two separate movies. Does Personal Video Database have a convenient way of handling these trailer files? How should I name the files so they are appropriately recognised.

Last of all. Has writing cataloguing data back to the movie files been considered, in a similar manner to ID3 tags are used for mp3 files. If .avi files do not support ID3 style tags them perhaps a separate file could be added to the directory containing the movie file, similar to how subtitles are handled with .srt files
The reason I ask is then the movie file would retain the cataloguing information enabling subsequent scanning to be much more accurate. Useful if you have your movie collection on a file server accessed by multiple computers and multiple user accounts, like I do 

[nostra]

1) Movie over more than one disc. It seams to identify "Shrek cd1.avi" "Shrek cd2.avi" etc as multiple discs for the same movie.

PVD identifies CD1..CDN and Title1..TitleN

2) Later movies in a series I usually put in the same directory (eg "Shrek 1.avi", "Shrek 2.avi", "Shrek 3.avi"). It seams to assume it is all one movie over multiple files so combines them. I'm not sure how I should name them to prevent this.

Episodes are not supported yet.

Also is there an efficient way of splitting up the enteries after they have been scanned (and falsely grouped)?

You can try to right click the movie and select "organize files by episodes". This function recognizes sXXeXX format.

3) Movies with a sample file / trailer eg "Shrek - sample.avi", "Shrek.avi" are assumed to be two separate movies. Does Personal Video Database have a convenient way of handling these trailer files? How should I name the files so they are appropriately recognised.

Such files are not supported yet. I'll add this to my TODO list.

Has writing cataloguing data back to the movie files been considered, in a similar manner to ID3 tags are used for mp3 files. If .avi files do not support ID3 style tags them perhaps a separate file could be added to the directory containing the movie file, similar to how subtitles are handled with .srt files
The reason I ask is then the movie file would retain the cataloguing information enabling subsequent scanning to be much more accurate. Useful if you have your movie collection on a file server accessed by multiple computers and multiple user accounts, like I do  Smiley

You can use export function to create such text files. Naming video files according to data from database is planned for future.

[patch]

Thank you for you wonderfully prompt response. You have been a great assistance.

I assume from

PVD identifies CD1..CDN and Title1..TitleN

Then to ensure sequels to movies are treated as separate movies I should ensure my titles do not inadvertently follow this format, for example I should use
"Shrek (2001).avi", "Shrek II (2004)" instead of
"Shrek 1 (2001).avi", "Shrek 2 (2004)"

Movies with a sample file / trailer eg "Shrek - sample.avi", "Shrek.avi" are assumed to be two separate movies.

Such files are not supported yet. I'll add this to my TODO list.

A similar issue occurs with movies I have two copies of (high and low resolution versions).
Or separate episodes of a TV serial which PVD has not recognised.

One way to facilitate suitable file associations would be to this would be to have an extra option in "Optimise Database..." which looks for separate entries with identical imdb link and asks how they should be linked (ie main movie, blue-ray/dvd versions, trailer, episode in series, or delete duplicate file)

Has writing cataloguing data back to the movie files been considered, in a similar manner to ID3 tags are used for mp3 files.

You can use export function to create such text files. Naming video files according to data from database is planned for future.

The function I'm trying to achieve is 100% rescan accuracy of "Scan for new movies/changed paths..." allowing fully automated rescan. I suspect to get the graphic images to rescan correctly I many need more than a text file but I suppose if it contained the Amazon link a text file would do.
(100% rescan accuracy is valuable when the movie files are on a central file server, accessed by multiple computers & users within the home. Also useful if file server structure is changed over time).

So my question is, do any of the output formats write a file to the same directory as each movie file, and will PVD then use this data when subsequently rescanning the file server from a new computer / user account.

[nostra]

Then to ensure sequels to movies are treated as separate movies I should ensure my titles do not inadvertently follow this format, for example I should use
"Shrek (2001).avi", "Shrek II (2004)" instead of
"Shrek 1 (2001).avi", "Shrek 2 (2004)"

Theoretically PVD will handle both version, but you can do this to be sure.
Title1..TitleN would mean:
"Shrek1 (2001).avi", "Shrek2 (2004)" without spaces

Quote
Movies with a sample file / trailer eg "Shrek - sample.avi", "Shrek.avi" are assumed to be two separate movies.
Quote
Such files are not supported yet. I'll add this to my TODO list.
A similar issue occurs with movies I have two copies of (high and low resolution versions).
Or separate episodes of a TV serial which PVD has not recognised.

One way to facilitate suitable file associations would be to this would be to have an extra option in "Optimise Database..." which looks for separate entries with identical imdb link and asks how they should be linked (ie main movie, blue-ray/dvd versions, trailer, episode in series, or delete duplicate file)

I'll consider implementing smth like this.

do any of the output formats write a file to the same directory as each movie file

No

and will PVD then use this data when subsequently rescanning the file server from a new computer / user account.

No

Could you describe your tasks and working environment more precisely, so I can imagine what changes need to be done.
I do not guarantee that all requested features will be available, but I am doing my best to suit most of the users.

[patch]

Could you describe your tasks and working environment more precisely, so I can imagine what changes need to be done.

With pleasure

Movie files are stored on a file server and accessed by multiple users / computers within the home.
PVD is an excellent way of selecting a movie to watch so I would like to use it on each computer on the network.
Setting up PVD on each computer requires "Scan folders for new movies" and pointing it at the file server.
Doing this the first time is labour intensive as I need to manually check each imdb and Graphics/Amazon link.

Doing it from subsequently computers could be fully automated (which would make PVD network friendly)
Probably the easiest way to achieve this is if PVD had "*.id3" export and import formats.

Export *.id3 would write a file to the same directory as each movie on the hard disk, using an identical name but .id3 file extension.
The id3 file would contain at least the imdb link, but preferably other generic fields (eg graphics or link to Amazon, but not individual user data such as their rating & if they have watched the movie).

During "Scan folders for new movies" PVD would then look for an .id3 file. If found it could add the movie without user intervention, making the "Scan folders for new movies" 100% accurate from a new computer or user account.

In an ideal world the id3 tag information could be appended to the movie data file, similar to mp3 music file. Unfortunately I suspect .avi does not support it http://msdn.microsoft.com/en-us/library/ms779636(VS.85).aspx and http://en.wikipedia.org/wiki/Comparison_of_container_formats I believe some of the later movie file formats support Metadata /Tags, so this may happen in the future, but in the mean time a separate file would achieve the same end.
Interestingly this program appears to put similar information within .avi files http://www.exefind.com/abcavi-tag-editor-P21011.html
I thought I read somewhere that it does it by putting the information within unallocated blocks within the file but I could be wrong. Actually I don't want to distract you as an external file is probably easier and would achieve the functionality I'm after.

BTW
id3 file format is not critical, I just choose it as it is relatively standard and currently stores similar information. A .xml files would probably work just as well.

[nostra]

OK, I understand.

One question: would it not be much better and easier if you install a database server on your Server PC and have a single PVD database on the server as well that is accessed by all the users in the network.

I plan to implement this in the next major update and it is much easier for me to implement than id3-like export/import.

P.S. the server can run on windows or linux. (Firebird)

[rick.ca]

Has writing cataloguing data back to the movie files been considered...

When I first read this, I found the idea appealing. It would offer greater peace-of-mind PVD would not be "confused," should I move or rename movie files. I'm trying out Media Center, and one of my disappointments is I cannot import PVD data into it. Probably just as well—I don't want to be maintaining two databases. But if PVD could write back to the movie files, I suspect MC would read the information automatically. In stark contrast, I've always maintained my music information in ID3 tags, and all of it is seamlessly available in whatever software I choose to use.

I'm not familiar with the tagging convention, but a lot of meta information can be saved in AVI files. I imagine the first issue here is MediaInfo (the DLL PVD uses to get information from movie files) only reads, does not write information. So nostra would have to figure out how to do that. And what about other file formats? While writing only to AVI's for this limited purpose may not be difficult, it opens the door to what could be a major new feature that would probably not be easy to support: What about other programs using the same information? Being able to get my PVD information into MC would be great, for example, but what would PVD do when MC wrote its version of the information back to the tags?

Movie files are stored on a file server and accessed by multiple users / computers within the home...

I suppose each user would want to maintain their own database, but based on the same store of movies on the server. Nevertheless, would it not make sense to built a "master" database of all movies on the server? Then each user could start with a copy of that, and delete the movies they're not interested in.

Probably the easiest way to achieve this is if PVD had "*.id3" export and import formats.

I thought ID3 was for music only. Much of the information is the same, but does it provide for tags unique to movies? I suppose a tagging convention—if that's what it is—doesn't necessarily address the issue of how such tags are included in various file types. Thus your suggestion the information could be saved to separate files. I've never come across anything, however, that would suggest other programs could make use of information saved this way. Even if this works for your specific purpose, saving the information in the movie files removes the risk of the files getting separated.

[patch]

One question: would it not be much better and easier if you install a database server on your Server PC and have a single PVD database on the server as well that is accessed by all the users in the network.

A server version would definately help with some but not all of what I'm trying to acheive.
It would allow multiple users to access the PVD on the file server 

The difficulties I would have are:
1) The "File server" in my case is just a PC with large hard disks which are shared over the network. People log in to multiple accounts to use it as a PC, so it would not be easy to ensure the server version of PVD was running, and running only once.

2) File storage updates. I regularly need to buy more hard disk space, which often results in the path to some of the movie directories changing. If the basic movie description data is stored with the movie file then PVD rescan is virtually garanteed to accurately update it's database (even if I needed to open a new database & scan in the file server again). With a server version I'm not as confident server configuration updates would go as smoothly.

3) When we go on holidays it is nice to bring a laptop +/- an external drive with some movies. If PVD supported tagging I could effectively bring it too, as setting it up for the new configuration would be trivial.

To illustrate how it could work, I can describe how audio is handled
I digitise my music, tag it, and put it on the file server (resulting in key track data being stored with the sound data).
Media Monkey http://en.wikipedia.org/wiki/Mediamonkey is configured to rescan the file server directories on startup.
So any new music is available to everyone at home.
As each user runs their own copy of Media monkey, they also have personalised data such as ratings, play lists, last played & frequency etc. (Some users even still persist with iTunes or Winamp)

I would like to achieve a similar thing with my movies, hopefully through PVD.

What about other programs using the same information? Being able to get my PVD information into MC would be great, for example, but what would PVD do when MC wrote its version of the information back to the tags?

I do not think movie file tagging is widely used or standardised yet (which is why we are having this discussion). As such I think it is very unlikely any tags created by PVD would automatically work with MC. However I believe tagging will be more widely used in the future, and at least a de facto standard will emerge with time.

I suppose each user would want to maintain their own database, but based on the same store of movies on the server. Nevertheless, would it not make sense to built a "master" database of all movies on the server? Then each user could start with a copy of that, and delete the movies they're not interested in.

Master databases worry me. Simple single user programs accessing movies on a file server sounds less complex and mor flexible to me.

I thought ID3 was for music only
..
saving the information in the movie files removes the risk of the files getting separated.

Agree
ID3 was suggested as a format to start from that much of the industry is familiar with.
Incorporating movie tags within movie data files will no doubt be the ultimate standardised solution. The separate file was a simple way of achieving the same functionality with all movie file types without compromising compatibility with movie players or needing to write to large video files. Yes the files would need to be copied together if movie files were moved, but I already do that with subtitled movies. Tags within movie files (.avi .mkv .wm etc) could be progressively added later as resources and tools permit.

[rick.ca]

I do not think movie file tagging is widely used or standardised yet...

That seems to be the case. I just spent hours trying to get image tag information into MC—it seems hopeless. I recall abcAVI Tag Editor saves IMDb information in AVI's, but I suspect the tags are the author's creation, and he's just writing them as a block into the file. So maybe the answer is it's easy to write information into AVI's, it won't affect other software that needs to read them, but there's little chance the information could be shared in any way.

Master databases worry me...

I was assuming this would be a one-time thing—just for getting started. I meant "master" only in the sense it would include all movies on the server, and users would start their own database using a copy of it.

I suppose your vacation scenario might be an example of where this functionality is required on an ongoing basis. But I wonder about that too. Wouldn't you just copy your database to the laptop, and then run scan folders for changed paths, pointing to the new file location? Yes, you want 100% accuracy, but in this case, the program only needs to check for changed paths—not new files. If it's trying to match the title directly to the new path while ignoring the filename already recorded, I'd say the code needs to be revised to take the old filename into account. I can't imagine why anyone would move files and change file names and expect PVD to rematch everything all at the same time. Furthermore, even though you can't drag a server with you on vacation, there's no reason not to take your entire database. This would be much easier and faster than rebuilding a new database. PVD will do a good job of telling you what's available to watch, and what you left behind.

Yes the files would need to be copied together if movie files were moved, but I already do that with subtitled movies.

I have no doubt you can do it—you're the one running an industrial network in your home! Other users don't where or how there files are stored. For them, separate files increases the risk of things going wrong. But then they're not likely to be interested in this feature, are they?