AllMovie+ script: Including data for custom fields

[rick.ca]

The AllMovie site has been replaced by AllRovi. See AllRovi movie script.


Attached is my modification of nostra's AllMovies.com script to include additional data for custom fields. Essentially, I changed the script to gather as much data as possible from the left-hand column of the "Overview" page. At the beginning of the script, I've documented the additional custom fields necessary to accommodate this data (see below).

Warning: This script uses the standard Category and Tags fields. Ensure your Field overwrite settings are set correctly to avoid conflicts.

All of the 17 custom fields included are optional. Just add the ones you want to Preferences - Movies - Custom items. There is no need to modify the script for fields you choose not to use. The script refers to custom fields by name, so if you use a different name, you will have to change it in the script.

I've done this largely by copying and adapting parts of nostra's script without understanding exactly how or why it works.  So suggestions or comments are welcome. Questions too—I love to share my confusion.

Here's the documentation:

Code: [Select]

CUSTOM AND NON-STANDARD FIELD MAPPINGS

 All custom fields are optional. All standard fields can be suppressed using
 Preferences-Plugins-Overwrite fields. If custom names are changed in PVD, they
 must also be changed where they occur in the body of this script. The type of
 custom field used is also somewhat a matter of personal preference. A list which
 is for viewing only will do just as well in a long text as a multiselect field.

ALLMOVIE CAPTION PVD FIELD FIELD TYPE

Cast Tab Cast memo
Production Credits Tab Production Credits memo
Review Tab Review memo
Run time Original duration number
Work rating AllMovie rating rating
Attributes Attributes multiselect list
MPAA + MPAA Reasons MPAA standard (IMDb is primary source)
Category Work type select list
Color type Color type select list
Types Types multiselect list
Flags Flags multiselect list
Keywords Keywords multiselect list
Themes Category standard  (sole source)
Tones Tags standard  (sole source)
Moods Moods multiselect list
From book (or story) Book short text
Set in Set in multiselect list
Box office Box office standard (IMDb is primary source)
Produced by Studio standard (IMDb is secondary source)
Release (or Premiere) Released short text
Released by Released by short text
    (Date)     Updated        short text
Current version attached:
09/24/2010 – 0.1.2.2 – Modified ParseCast procedure to save entire Cast tab to custom memo field, as well as individual actors.

[attachment deleted by admin]

[rick.ca]

Now adds current date to Updated field.

The updated script is attached to my original message above.

[darichman]

Rick, this is great - thanks for sharing.

Just wondering, do you mainly use allmovies for your metadata? I was trialling a system with custom fields a while ago to attempt to maintain both IMDb and AMG fields in the same database (Used standard fields for IMDb and a whole bunch of custom fields for AMG - AMG Director, AMG Synopsis etc etc). It was getting a bit unwieldy.

There are a few situations where AMG is incomplete compared to IMDb or vice versa... what's your system?

[darichman]

Some feedback... works great! A few little niggly bits I've noticed so far...

1. Box Office doesn't format correctly in the standard field. Works fine as a "short text" custom entry

Code: [Select]

//Box office
 AddCustomFieldValueByName('Box Office', HTMLValues(HTML,
                             '<span>Box Office</span>', '</table>',
                 '<li>', '</li>',
                 ', ', EndPos));
2. Your MPAA + Reason formatting works okay, but if there is no reason listed, you get an output of "Rated M for ." etc

3. I added support for the "Region" field in AMG. It's being used more and more frequently on the site, but many older movies don't have it populated.

Code: [Select]

//Region
 AddCustomFieldValueByName('Region', HTMLValues2(HTML,
                             '<span>Premiere</span>', '</table>',
                 '<a href="', '</a>',
                 ', ', EndPos));
4. I seem to get a rating of 10 for every movie... must have a closer look.

Also, I'm not sure if it was you or Nostra who added the Reviews and Producion Credits functionality, but that's awesome - thanks

[rick.ca]

Rick, this is great - thanks for sharing.

You're welcome. It was painful, but I figured it was my turn.

Just wondering, do you mainly use allmovies for your metadata?

I use both IMDb and allmovies together—usually run in a batch. IMDb goes first, because it is generally more complete. I let it handle all the people. You can see from my documentation where I use allmovie data to overwrite standard fields which may have been populated by the IMDb plugin. The rest goes into custom fields. As you probably know from doing this yourself, my approach in modifying the script was just to grab whatever I could from the stuff listed in the right-hand column of the page. I do actually have all those custom fields in my database, but only for the sake of being complete. The data elements I appreciate most are the Synopsis and Review (my original reason for going to allmovies), Tones (which I find gives a really good indication of how a movie feels), and perhaps Attributes. The rest is nice to have, but not critical.

My preferences and my skin design probably feed off each other. With that in mind, I'll attach some screenshots so you can see what I mean. We don't yet (as of 0.9.9.4) have the ability to change the placement of custom fields—they're still just dumped into the custom section of the skin. This is why I'm very deliberately placing some allmovie data into standard fields—so I can place them where I want them in my skin. Also, you may note most of the list data goes into Multiselect list fields, even thought they may look better as List or String fields. This is because I don't care so much about looking at them—I'd rather have them take up less space, and be used primarily as a means of grouping.

When I was doing this before, I suppose it got a bit unwieldy. Redoing the script, however, forced me to rethink what I was doing and why. Now it seems to work quite well. I do have to make sure my Overwrite field settings are correct—and stay that way. That can be nuisance when a plugin or script is used for more than one thing (e.g., movies and series). BTW, I think I have nostra partially committed to implementing my fix for that in version 2.0.

[attachment deleted by admin]

[darichman]

Thanks, that was really helpful... I agree that it's nice to know where the data came from. Up until now, I've been using IMDb for all standard fields, with AMG fields all belonging to custom fields with a "AMG " prefix. It's hard to get the best of both worlds... custom fields don't work too well with "People" fields...

A template to decide which order fields are displayed in would work really well, for both standard and custom fields I think...

[rick.ca]

A few little niggly bits I've noticed so far...

Thanks! My routine of downloading from two sources is concealing some bugs.

1. Box Office doesn't format correctly in the standard field. Works fine as a "short text" custom entry

Yes, I started with a custom field, but then decided I would prefer to use the standard. So I reverted to nostra's code. A simple dollar figure seems to work. But "$106,260,000/ Top grossing film of 1979" becomes "1,062,600,009" (!) Obviously the standard field is numeric. So I added the "+ 1" in the following:

Code: [Select]

//Box office
 curPos := PosFrom('<span>Box office</span>', HTML, EndPos);
 if curPos > 0 then begin
  curPos := PosFrom('<li>', HTML, curPos) + 5;
  EndPos := PosFrom('</li>', HTML, curPos);
  TmpStr := Copy(HTML, curPos, EndPos - curPos);
  curPos := Pos('/', TmpStr);
  if curPos > 0 then
   Delete(TmpStr, curPos, Length(TmpStr) - curPos + 1);     <-----
  curPos := Pos(':', TmpStr);
  if curPos > 0 then
   Delete(TmpStr, curPos, Length(TmpStr) - curPos + 1);     <-----
  AddFieldValueXML('money', TmpStr);
 end;
I don't really understand the code (imagine a monkey banging on the keyboard ), so I'm not sure if that's an appropriate modification, although it seems to work for ones I can find. Allmovies doesn't seem to have much box office data—which is why I decided against an additional custom field.

2. Your MPAA + Reason formatting works okay, but if there is no reason listed, you get an output of "Rated M for ." etc

Yes, just being lazy here. I only use this a backup to IMDb, which normally have exactly the same data (that's where the "Rated X for ..." format comes from). I couldn't be bothered fixing it for the rare case where IMDb would not have the data, and allmovie would have the rating but not the reason. Still, just fixing it would be easier than making excuses...

Code: [Select]

//MPAA Reasons
 TmpStr1 := HTMLValues(HTML, '<span>MPAA Reasons</span>', '</table>', '<li>', '</li>', ', ', EndPos);
 if (TmpStr1 = '') AND (TmpStr2 <> '') then
 AddFieldValue(mfMPAA, 'Rated ' + TmpStr2);
 if (TmpStr1 <> '') AND (TmpStr2 <> '') then
 AddFieldValue(mfMPAA, 'Rated ' + TmpStr2 + ' ' + TmpStr1 + '.');
...well, maybe not. I think that does it, but I'm not sure.

3. I added support for the "Region" field in AMG.

I think that the one that only occurred in 2% of my movies—so I deleted it. There are a number of such obscure items. The site design is very annoying. Tags like that are usually linked to a page with a definition of the term and/or a list of movies with the tag value, but there's no other way to access those pages. In other words, I have to know of a movie tagged with a region before I can find out anything about that tag. To enumerate a few examples...

854: From Play
855: From Poem By
856: From Poem
857: From Screenplay By
858: From Screenplay
860: From Story
861: Lampoons "Name" (?)
862: Location
862: Movie Budget
862: Recognizable Quote...

Sorry—thought I might find "Region" or something interesting if I kept going. I think they're on drugs, or just want to piss-off script writers.

If only we had the entire list of tags and they appeared on the page in tag number order (they don't), that would be a more reliable way to reference these items, and perhaps aggregate some of them into something useful. For example, the "From" ones could be combined into one field populated with "From book ____",  "From play ____", etc. )I combined Book and Story, but I couldn't find the rest.)

4. I seem to get a rating of 10 for every movie... must have a closer look.

Their rating tend to be generous, but that's not right. I don't know why this may cause you a problem, but note the script is using a custom field, rather than the usual (i.e., as in nostra's version) Other rating field.

I'm not sure if it was you or Nostra who added the Reviews and Production Credits functionality, but that's awesome

That would be nostra. I'm hoping you'll study his technique and use it to grab the "Similar Works" entries from the Overview page and save them as Connections. While it wouldn't be quite as useful, it would nevertheless be super-awesome! Besides, there's nothing left to grab, so this one's yours.

[darichman]

Oh wow, I've never seen some of those fields up there! haha

Yeah, I'd like that extra info too... I wonder how it would be best presented though - maybe in a note/memo field. I'll have a look - but my approach so far has been similar to yours... look at the existing script and trying to adapt it without breaking it. I've broken it a few dozen times now

[rick.ca]

I'd like that extra info too...

I figured analyzing the tag URLs would be a good way to figure out what information is there and how it's organized. See attached bookmarks file (remove the .TXT and open in browser and/or import bookmarks). I gave up after 1,555 (tag numbers up to 2,200). Still no sign of Region. 

Note many of the URLs have no information other than the name of the tag. Others have a brief description, definition or instruction. Many are to lists of movies with the tag—so the link to those can be useful. You know, when you just have to have direct link to movies made in Albania...

[attachment deleted by admin]

[rick.ca]

In an attachment there is a script which works both with www, and without www.

Applied same fix to this script (attached to first message above).

Thanks, Reset.

[rick.ca]

I have modified the script to conform to changes in the original provided with 0.9.9.5. It's attached to the first message above.

[rick.ca]

As mentioned here, I've modified the script to save "Run time" to the custom field OrigDuration instead of the standard Duration field. I now no longer download to Duration, leaving this for the actual media length according to MediaInfo (i.e., the program's scan of the media file). The revised script is attached to the first message above.

BTW, the actual name of the field "Duration" is Length—that's how it must be referred to in a skin.

[sunspot]

This modification just made my day! I love it. 

Does it work with any version of the allmovies.com plugin or does it need updating, too?

[rick.ca]

Welcome, sunspot. Glad you like it.

There is no AllMovie plugin. This is a modification of the script written by nostra (and included in the installation package). Downloading/importing from any particular source is effected by a plugin or a script, not both.

[sunspot]

Thank you for your answer! You seem to be a bedrock for this forum.

I obviously don't know enough to just ask a proper question. A follow-up one, if I may: So the script would need updating like any other, if the site allmovie changes? I ask to understand a little (little) bit better what's going on.

[rick.ca]

So the script would need updating like any other, if the site allmovie changes?

That's right. But, as you can see from this thread, it has required only a few minor changes. Please do let me know if you find any problems. It could be something has changed, but I haven't noticed because I'm no longer adding large numbers of new movies to my database.

[sunspot]

Oh, that's something I can provide - heavy duty importing¨

And I will certainly try to give meaningful feedback. But, at the moment, it's just perfect! Don't see anything broken or so. Quite the contrary, I love how it replaces the imdb tags with more meaningful ones. Tone and Mood (where available) are superb, too. Thanks again. I think I will head now to the donation page, if you will excuse me. 

[deazo]

 Hi Rick,

 This scripts looks very interesting and I'd like to get these extra fields too.
 Unfortunately, (and i'm a bit embarassed here... ) I can't get pvd to "recognize" the script. I must be doing something wrong.
 I have extracted it into the "script" folder in pvd's directory, and it does not show up in the "import" drop down in pvd.
 I even renamed it so that it would replace the allmovie original script and again it failed to show up.
 Can u help?
 
 Thanks a million.

[rick.ca]

Sorry, I forgot to update the script posted here. I've done so now. The version is now...

SCRIPT_VERSION = '0.1.1.9';   //Converted to use Unicode strings effective 0.9.9.15

[Ivek23]

What and how is necessary to instal AMG Work ID in the script.
That would do just for myself, of course, if it is not too much input in the script.

Sorry for the dumb question.