Export plugins for J River Media Center

[raldo]

I'm "working on" an export plugin for JRiver mediacenter.

Now, I think I have worked out a way of doing the file selection and metadata import "seamlessly" on the MC side.

Some questions, possibly on the SDK or commandline on the PVD side:
o Is there a way to start a *silent* scan on a single file from within the export plugin framework? I.e. is there a part of the SDK or the command line interface that facilitates this?
o Is it possible to somehow get the scan results programmatically so that the scan results can be resolved programmatically?

Thanks.

[nostra]

What do you mean with "scan"?

Is it possible to somehow get the scan results programmatically so that the scan results can be resolved programmatically?

You can retrieve any information field. I'll explain how when I am back home in the evening.

[raldo]

What do you mean with "scan"?

Scan: When you first have set up your "watch folders" and then select "scan folders" to retrieve metadata. (PVD isn't installed on this PC so I'm not completely sure what the exact wording is)

By "Scan results" I mean the disambiguation dialog that pops up when you import meta data. For example to select between the original "Open your eyes (abre los ojos)" and the Hollywood remake.

[nostra]

Hmm, currently there is no way of starting the scanner in silent mode and get it's results.
What do you need such functionality for? Could you explain exactly what the end result should look like?

[raldo]

Good idea.

I'm thinking of making an JRMC-PVD interface where PVD serves metadata to JRMC.

My suggestion below is for a "seamless interface" where no action needs to be taken on the PVD side.

The way it would work (if possible) is as follows:
(i) In MC, select one or more files and select "Import metadata from PVD"
(ii) The MC plugin signals PVD sequentially for each filename with with a request for metadata.
(iii) A receive thread on the PVD side processes each request and initiates a metadata scan.
(iv) When the scan for a file has finalized, PVD signals back to MC, either a disambiguation list of alternatives (as descibed above) or the "one and only result"
(v) If MC receives a list of alternatives, the MC plugin brings up a dialog from which the user can select the correct item. Then, MC requests data from PVD.
(vi) When the MC plugin has received data, it populates fields in the MC db according to user configuration in the MC Plugin.

Communication would take place via UDP.

Alternatively, MC could retrieve the data from the database (using the API some other user provided in "some other thread").

[rick.ca]

My suggestion below is for a "seamless interface" where no action needs to be taken on the PVD side.

I take it your thinking is media may be first imported to MC, and therefore the plugin needs to be able to induce PVD to create a record and download the metadata. Slick, but I think there are too many things that could go wrong. Even if it does work, the plugin is more likely to need updating when PVD (the program) is updated. But most importantly, this doesn't make sense from a work flow point-of-view. If PVD is being used to collect and manage metadata for videos, then it should be allowed to do that—with user supervision—before the data is imported to MC. Otherwise, the data may be incomplete or erroneous. To make matters worse, it will remain that way in PVD until the user recognizes this is so.

Alternatively, MC could retrieve the data from the database

It seems to me this has to be much simpler, and doesn't violate the work flow premise. It would be reasonable to assume that, whatever the state of the data at the time the plugin is run, it's appropriate to import it. Also, rather than requiring the user to select items in MC to be updated, the plugin could automatically update items based on PVD's Date updated (new in 0.9.9.5). There would need to be an equivalent field in MC that the plugin updates. Knowing the MC user can choose to use Auto-import or not, it makes sense to ignore records that exist in PVD but have not yet been added to MC. Such records would be updated after they have been added to PVD.

BTW, we discussed in the MC forum how perhaps credits can be handled using two fields in MC. For example, for actors, import [Name] to Actors and [Seq#]. [Name] as [Role] to a second field (Actor credits?). Are you planning on building that into your plugin? I don't see a special-purpose credits field being added to MC, so this is probably the way to go.

[raldo]

I take it your thinking is media may be first imported to MC, and therefore the plugin needs to be able to induce PVD to create a record and download the metadata. Slick, but I think there are too many things that could go wrong. Even if it does work, the plugin is more likely to need updating when PVD (the program) is updated. But most importantly, this doesn't make sense from a work flow point-of-view. If PVD is being used to collect and manage metadata for videos, then it should be allowed to do that—with user supervision—before the data is imported to MC. Otherwise, the data may be incomplete or erroneous. To make matters worse, it will remain that way in PVD until the user recognizes this is so.

Yes, I suppose things could go wrong, especially the first time you export from PVD to MC. But you could still browse the received data in MC. I'm not saying that we don't *need* the PVD GUI, I'm just saying that, hopefully, MC's theater view browsing will improve over time...

But after that, when you do one movie at a time, detecting errors should be easier..

It seems to me this has to be much simpler, and doesn't violate the work flow premise. It would be reasonable to assume that, whatever the state of the data at the time the plugin is run, it's appropriate to import it. Also, rather than requiring the user to select items in MC to be updated, the plugin could automatically update items based on PVD's Date updated (new in 0.9.9.5). There would need to be an equivalent field in MC that the plugin updates. Knowing the MC user can choose to use Auto-import or not, it makes sense to ignore records that exist in PVD but have not yet been added to MC. Such records would be updated after they have been added to PVD.

This seems to be a "phase 2" issue. Remember, I haven't even started writing code yet I am not even sure yet if it will be possible to do my outline from above...

BTW, we discussed in the MC forum how perhaps credits can be handled using two fields in MC. For example, for actors, import [Name] to Actors and [Seq#]. [Name] as [Role] to a second field (Actor credits?). Are you planning on building that into your plugin? I don't see a special-purpose credits field being added to MC, so this is probably the way to go.

I haven't really "solved" yet how field mapping should be done. I think that Nostra's excellent system of field mapping/parsing could be used as is (see, for example, plain text export). The advantage of this is that it has already been done and the parsing engine is designed around the database. The parsed output, one per file, would then be transmitted to MC via UDP. MC would then simply stick the field mappings into the proper MC fields.

The plugin on the PVD side would then be installed together with a default template (with a "pages template" for Credits which would surely accomodate your sequencing request).

[rick.ca]

I haven't even started writing code yet

I understand, and I hope you don't mind this kind of input at this stage.   I assume it's preferable to design, then code. At the same time, I do understand design ideas often have to radically change when you're doing the coding and discovering what actually works and what doesn't. So please consider my comments as having a helpful intent, in the spirit of "talk is cheap."

Yes, I suppose things could go wrong, especially the first time you export from PVD to MC. But you could still browse the received data in MC. I'm not saying that we don't *need* the PVD GUI, I'm just saying that, hopefully, MC's theater view browsing will improve over time.

The point I'm making has nothing to do with which GUI is preferred. It's that the critical database management function of ensuring the completeness and accuracy of the metadata is being performed using PVD. Once I've done that, I don't want it undone by the MC plugin adding more records (with a significant likelihood of inaccuracy or incompleteness) I don't "know" about. Yes, it's possible to detect errors in MC, and it's possible to determine what records the plugin has added to PVD (so I can check them for completeness and accuracy). But why would I want to waste time doing so? Why would I want to give up the peace-of-mind of simply knowing the metadata in MC is exactly the same as that which I ensured was complete and accurate in PVD?

Perhaps few users take data integrity as seriously as I do, but I really believe my point is far from an academic one. Maintaining some degree of data integrity in one database is challenging enough. Doing so with two databases is that much more difficult. There's not much that can be done about the two being out-of-sync because of timing differences in the importing of media files (including the possibility one or the other import routines fails for a particular file). If MC is being updated using the PVD database, the user has a known point of reference, can assess the results, and deal with those matters. If the plugin is adding items, however, the point of reference is lost, and the user has no way of telling what the difference is, or whether it matters. Once users see their data is out-of-sync and they can't determine why (or, more likely, just become confused), they will lose confidence in the plugin.

This seems to be a "phase 2" issue.

What do you mean—the ability to update only changed records?

I was assuming a reasonable "phase 1" would be simply importing directly from the database data for matching media files in MC, and things that might require the plugin to interact with PVD would be "phase 2."

The plugin on the PVD side would then be installed together with a default template (with a "pages template" for Credits which would surly accomodate your sequencing request).

Yes, I can see that working, but it seems to be a very indirect method of accessing information in an open SQL database. Are there not software "tools" you could use that would provide the core functionality you need? I'm sorry—I don't have a clue what I'm talking about—it just that pulling data from a database to put in another seems to be such a "generic" requirement.

Regardless of the technical challenges, I think there is a more serious problem with this. Having to configure a template is not very "user-friendly." Those of use who are used to this system (who are also MC users—all 4 of us?) could handle this. For MC users coming to PVD for a video metadata solution, however, I fear it adds too much to the learning curve. Also, it would make the whole exercise not that much different than the current export/import solution. I was imagining what I consider to be a fairly common field mapping interface. In it's simplest form, it would display all the fields (standard and custom) found in the database in one column. In a second column, drop-down boxes would be used to select from the fields available for mapping the data to. A fancier version would display field types and assist in the selection of a compatible MC field type, or other types the plugin is able to convert the data to. It might also offer default mappings, and allow the adding of new MC fields "on-the-fly."

For the MC user coming to PVD, the experience I'm hoping for is one where they can focus on learning PVD and building a database, without concern about how the information is going to get into MC. When they're done, the plugin will provide them with a user-friendly means of mapping and importing the data, regardless of what information their database contains or how they want it imported to MC. Furthermore, I think it reasonable to assume—especially in early days—users will want to make regular changes to their configuration. They should be able to do so directly in the plugin interface, at any time.

So, this has been my very long-winded way of suggesting... Maybe the simplest and best solution is one which pulls the data directly from the PVD database, and maps it to MC fields based on a flexible (map anything to anything) configuration interface.

[patch]

I'm "working on" an export plugin for JRiver mediacenter.

Now, I think I have worked out a way of doing the file selection and metadata import "seamlessly" on the MC side.

Why not emulate what is done for electronic music.
Music files are tagged, the taggs are written back to the music files, other programs can then read the metadata.
Music files often use id3 or similar. Unfortunately mediainfo.sourceforge.net will not support tag writing in the forseable future, so we won't be able to do it that transparently. http://sourceforge.net/forum/forum.php?thread_id=1695277&forum_id=297609

What we could do is get PVD to write text data to a tag file located near the movie file (and with the same name). This could be a .txt or better still a .xml maybe easier to maintain. Image data could be written to a similarly named & located .jpg or similar file.

Once written this data could then be read into another program such as JRiver mediacenter or another instance of PVD.

The main advantage of using a relatively standardised tagging format is changes can be made on either side with relative independence. Also the interface is likely to have broader applicability.
http://www.videodb.info/forum_en/index.php?topic=1002.0
http://www.videodb.info/forum_en/index.php?topic=870.msg2516#msg2516
http://www.videodb.info/forum_en/index.php?topic=1249.0

[rick.ca]

What we could do is get PVD to write text data to a tag file located near the movie file (and with the same name). This could be a .txt or better still a .HTML maybe easier to maintain. Image data could be written to a similarly named & located .jpg or similar file.

I appreciate you have other reasons for wanting to see something like this implemented, but I don't think it's a sensible solution for what's being discussed here. Raldo is considering creating a plugin for J. River Media Center that would pull data directly from the PVD database into MC. That will be far more efficient, reliable and easier to maintain than writing a text and an image file for every video file. True, it will have no relevance for other applications, but MC users won't care about that—they're using MC to manage and view videos. This plugin has the potential to bring a portion of a large and sophisticated user base to PVD, but I believe it has to be something which serves their needs—without compromise.

Maybe Reset would be willing to advance your idea. We already have the script for getting information from a text file. He's going to do a plugin for importing images "beside" the video file. Maybe he'd be willing to expand that project to include importing and exporting images and text. That would have of value as a versatile utility function, as well as, hopefully, serving your need.

[patch]

The way it would work (if possible) is as follows:
(i) In MC, select one or more files and select "Import metadata from PVD"
(ii) The MC plugin signals PVD sequentially for each filename with with a request for metadata.
(iii) A receive thread on the PVD side processes each request and initiates a metadata scan.
(iv) When the scan for a file has finalized, PVD signals back to MC, either a disambiguation list of alternatives (as descibed above) or the "one and only result"
(v) If MC receives a list of alternatives, the MC plugin brings up a dialog from which the user can select the correct item. Then, MC requests data from PVD.
(vi) When the MC plugin has received data, it populates fields in the MC db according to user configuration in the MC Plugin.

Communication would take place via UDP.

Alternatively, MC could retrieve the data from the database (using the API some other user provided in "some other thread").

Here I agree with rick.ca, PVD has put a lot of effort into collecting movie data from various sources so let it do it's job as it was designed to. Implementation options
a) The simplest is get the user to run PVD first.
b) MC plugin requests data from PVD, then PVD use its full GUI to retrieve the data prior to returning it to MC plugin
c) Combine the above. MC plugin request data from PVD, if know to PVD it returns full information, if it is not in the PVD data base it returns what it knows (ie nothing).


Looking at the user work flow.
User loads movies into PVD, pulling metadata from various sources
User Quits PVD (firebird doesn't allow multiuser / multiprogram access in PVD).  This assumes Media center plugin accesses PVD database directly. If piping between plugins the PVD would need to be left running.
User runs J. River Media Center
User runs plugin to pulls data from PVD/firebird database for all available or recently updated records

If intermediate tags were used the work flow would be
User loads movies into PVD, pulling metadata from various sources
User runs script / plugin in PVD to export tags for all or recently updated (I would like PVD to do this transparently eventually, maybe via an option to "Maintain tags")
User runs J. River Media Center
User runs plugin to pulls data from tags for all available or recently updated records (could be automated via a "Scan folders on start up option")

Other differences
The tag files approach results in
a) Multiple tag file being stored on the hard disk (which some users may object to).

The J. River Media Center plugin approach
a) Would probably require updating each time PVD or J. River Media Center was updated.
b) Implies low level access to the firebird database is available. Alternatively if implemented as matching plugins in PVD & J. River Media Center, then we need a pipe function between plugins transfering a data format which will probably be similar to the "tag approach". Reading tag data from J. River Media Center plugin via PVD plugin via firebird may be slower than pulling it directly from windows file system tags depending on incremental update optimisation done.

I don't think it's a sensible solution for what's being discussed here. Raldo is considering creating a plugin for J. River Media Center that would pull data directly from the PVD database into MC.

Fair enough.
We agree aiming for a no compromise solution is appropriate. We differ on what level of abstraction is ideal

[raldo]

Nostra,

Are you interested in integrating PVD with MC as I outline above, considering that there will be needed sw changes on the PVD side?

On the other hand, do you think the method I outlined above is "good"? Is there maybe another way in which this can be made seamless?

[nostra]

It is very difficult for me to say which method is the best as I have never worked with MC and it's API.

If I would need existing data from PVD I would just query the PVD database directly using SQL and then put data into MC using it's API. This seems like most powerful and universal way to me.

Another approach that is much easier is to create an export template for PVD and write an import plugin for MC that understands files generated using the template. 


If i would need to get data from MC into PVD I would just write an import plugin that uses MC API.


I have no problem adding some kind of communication protocol in PVD if it would make sense, but I do not see a need for this at this point.

[raldo]

It is very difficult for me to say which method is the best as I have never worked with MC and it's API.

Well, I was more thinking in general terms. Would you view this as a good "general method" of integrating with a front end type of system?

If I would need existing data from PVD I would just query the PVD database directly using SQL and then put data into MC using it's API. This seems like most powerful and universal way to me.

Another approach that is much easier is to create an export template for PVD and write an import plugin for MC that understands files generated using the template. 

If i would need to get data from MC into PVD I would just write an import plugin that uses MC API.

Some other users in your forum have already solved this. I.e. we can Import metadata from PVD.

But it requires several manual steps and that's what I'm trying to "fix" with the solution outlined above. I.e. what I'm trying to achieve, is a tighter integration between MC and PVD which doesn't require several steps and switching back and forth between applications after the user has acquired and imported new movies in MC.

I have no problem adding some kind of communication protocol in PVD if it would make sense, but I do not see a need for this at this point.

Fair enough, It's your call..

[rick.ca]

If I would need existing data from PVD I would just query the PVD database directly using SQL and then put data into MC using it's API. This seems like most powerful and universal way to me.

Raldo, I don't understand your apparent reluctance to consider this may be the best solution.

[raldo]

If I would need existing data from PVD I would just query the PVD database directly using SQL and then put data into MC using it's API. This seems like most powerful and universal way to me.

Raldo, I don't understand your apparent reluctance to consider this may be the best solution.

Well, data retrieval is really not the issue at this point. As I've said before, I'm interested in a system where the entire operation of metadata import can be done from MC.

This also involves triggering a search for new data and resolving ambiguities in the results.

--------------

Btw, as I explained above, one issue with going directly at queries, is that one somehow has to translate a user configuration into queries via code. This mechanism already exists within Nostra's parsing language. Thus my apparent reluctance.

[rick.ca]

I'm interested in a system where the entire operation of metadata import can be done from MC.

I don't think the benefits that might be had using this approach would come close to outweighing the data integrity issues I raised above. I wouldn't risk messing-up my database using it, even if it worked. And I'm sure it won't. It's already enough of a challenge controlling "the entire operation of metadata import" directly from PVD.

one somehow has to translate a user configuration into queries via code

I don't understand. Surely the database can be queried to determine the field names and types. And then all the data for records with a pathname can be extracted. I'm sure there may be complications I can't imagine for a process of querying the database one record at a time and updating the corresponding MC record. But if this is impossible, couldn't the plugin still extract all the data temporarily to XML and import that? It's been a while since I've done such an import to MC, but it seems to me you don't even have to worry about whether or not the media files or fields exist in MC—if they don't, they'll simply be ignored.

[raldo]

I'm interested in a system where the entire operation of metadata import can be done from MC.

I don't think the benefits that might be had using this approach would come close to outweighing the data integrity issues I raised above. I wouldn't risk messing-up my database using it, even if it worked. And I'm sure it won't. It's already enough of a challenge controlling "the entire operation of metadata import" directly from PVD.

Yes, but one could still handle data integrity issues with a seamless system. You mentioned a new MC field "DateImportedFromPVD" which could be used to block import. Or some other mechanisms could be used.

It's not just that, though: Some users may have "lesser criteria" on data. For example, some of my friends use MC but complain about the lack of metadata possibilties for movies. Or my sister imports some movies into MC and her husband has no idea what the movies are about. A description, year, director might be nice for them.

But none of them are ready for PVD in its current state.

couldn't the plugin still extract all the data temporarily to XML and import that?

Yes, that's exactly what I am saying in the above posts. Use the mechanisms that are already there. (but pass the data through UDP instead)

[rick.ca]

But none of them are ready for PVD in its current state.

If that's the design premise, it's hard to imagine how your plugin will further the development of PVD. It would make more sense for nostra to give you his code so you can incorporate the acceptable parts of it into your MC plugin.

[raldo]

If that's the design premise, it's hard to imagine how your plugin will further the development of PVD.

Why? You could still use PVD in the way you've always used it and prune the data. And then use an MC plugin for import.

The difference would be that more people would use PVD.