AllRovi movie script

[RazorHall]

"Could not compile script: AllRovi.psf."  I don't get it.  The only change I made was moving the AMG ID section and putting it exactly where Ivek has it in his post.

[rick.ca]

The log normally indicates the error preventing the script from being compiled.

But please try version 13. I believe I solved the issue. I've not much interest in figuring out why your changes to a previous version don't work. I have a hard enough time understanding why my latest version works. Or doesn't.

[RazorHall]

Well, I actually didn't expect you to have any interest in it, since it was something Ivek had suggested, and I was really waiting for him to post a possible solution when he got around to it.  I also didn't realize you'd created a new version when I wrote my previous post, or I wouldn't have wasted time writing it before trying the new version.   But yes, with the latest script those rare titles that weren't getting Characteristics information before are now working correctly.  Thanks!  I'll let you know if I come across any other problems.  By the way, Ivek mentioned earlier in the thread that certain movies at AllRovi have a "More" button, and the program currently isn't downloading those themes shown under "More".  What are the chances the script will eventually be able to get those too?

[rick.ca]

I'll let you know if I come across any other problems.

Please do. I'm particularly interested in what my attempts to fix things have broken elsewhere.

By the way, Ivek mentioned earlier in the thread that certain movies at AllRovi have a "More" button, and the program currently isn't downloading those themes shown under "More".  What are the chances the script will eventually be able to get those too?

I might take a shot at that. It doesn't matter to the script, but notice Rovi's treatment seems to be inconsistent. Some records use the 'More' button, others let the buttons wrap to a second line. No, wait—I saw one with a 'More' button on a second line. Maybe it's an editorial decision that the number shown are enough.

[rick.ca]

Another day, another version...

Version 14 attached (and to top post). Changes:

• changed 'SavedPos' variables to single reference point: 'topPos'
• ...use 'topPos' if required data not found from 'endPos' previously set
• changed 'Themes' to include values behind 'more' button
• changed 'Themes' and 'Characteristics' to remove capitalization of prepositions and conjunctions
• changed 'Characteristics to remove 'Attributes' and 'Moods'

[attachment deleted by admin]

[Ivek23]

1. )

Code: [Select]

//Genres  
 //modified by rick.ca 07/30/2011 to add custom field
 endPos := topPos;
 TmpStr := HTMLValues(HTML,
'<dt>genres</dt>', '</dd>',
'">', '</a></li>',
',  ', endPos);
 AddFieldValue(mfGenre, TmpStr)
 AddCustomFieldValueByName('Genres', TmpStr)
This code does not work for me - does not transfer data info.


2. )

Code: [Select]

//Genres  
 //modified by rick.ca 07/30/2011 to add custom field

 TmpStr := HTMLValues(HTML,
'<dt>genres</dt>', '</dd>',
'">', '</a></li>',
',  ', endPos);
 AddFieldValue(mfGenre, TmpStr)
 AddCustomFieldValueByName('Genres', TmpStr)
While this works for me - the correct data transfer standard info box in the custom field is doubling the first word (example: Drama, Drama, Romance).



3. )

Code: [Select]

//Genres  
 //modified by rick.ca 07/30/2011 to add custom field
 
 TmpStr := HTMLValues2(HTML,
'<dt>genres</dt>', '</dd>',
'<li>', '</li>',
',  ', EndPos);
 AddFieldValue(mfGenre, TmpStr)
AddCustomFieldValueByName('Genres', TmpStr)
This is working properly - the correct data transfer standard info box and in the custom field (example: Drama, Romance).

[RazorHall]

I haven't done a lot of testing yet, but have noticed a couple things...

First, in cases where there's a "More" button for themes, the script seems to be getting the first theme under "More" but not any others listed below it.  A step in the right direction, though!  

Also, themes for certain films aren't being downloaded.  A few examples are Dark Reality, Street Vengeance, and Hot Target.

[Ivek23]

First, in cases where there's a "More" button for themes, the script seems to be getting the first theme under "More" but not any others listed below it.  A step in the right direction, though!

This one

but not any others listed below it

is not true for example 10 Things I Hate About You (1999) pass this

Similarity, High School Life, Love Triangles, Opposites Attract, Schemes and Ruses, Cliques, New Kid in Town, Sibling Relationships

in this code

Code: [Select]

//Themes (Category)
 //modified by rick.ca 07/09/2011
  TmpStr1 :='';
  TmpStr2 :='';
 TmpStr1 := HTMLValues(HTML,
'data-typeOp=', '</div>',
'">', '</button>',
', ', endPos);
 TmpStr2 := HTMLValues(HTML,
'<div class=''morebox''>', 'div></div>',
'''>', '</div>',
', ', endPos);
 if TmpStr2 = '' then
  TmpStr := TmpStr1
 else
  TmpStr := TmpStr1 + ', ' + TmpStr2;
 TmpStr := StringReplace(TmpStr, 'More, ', '', true, true, true);
 TmpStr := StringReplace(TmpStr, ' A ', ' a ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' An ', ' an ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' And ', ' and ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' & ', ' and ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' By ', ' by ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' For ', ' for ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' From ', ' from ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' In ', ' in ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' Into ', ' into ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' Of ', ' of ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' On ', ' on ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' Over ', ' over ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' The ', ' the ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' To ', ' to ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' Under ', ' under ', true, false, true);
 TmpStr := StringReplace(TmpStr, ' With ', ' with ', true, false, true);

 if GET_THEMES then
AddFieldValue(mfCategory, TmpStr);
 AddCustomFieldValueByName('Themes', TmpStr)
However, in the custom field with the same code like this

Similarity, High School Life, Love Triangles, Opposites Attract, Schemes and Ruses, Schemes and Ruses, Cliques, New Kid in Town, Sibling Relationships

underline is doubled.

[RazorHall]

This one is not true with this code

But that's not an official part of the script yet, is it?  So I'd have to go in and replace that section every time a new version was released, yes?  I also don't like that it puts "Similarity" among the themes.

[Ivek23]

This one is not true with this code

But that's not an official part of the script yet, is it?  

It is true that this part of the revised Rick.ca code, since the earlier part of his code, which was transferred there last theme is the "more" button.

So I'd have to go in and replace that section every time a new version was released, yes?

Unfortunately yes, because otherwise you script will not work exactly right.

 I also don't like that it puts "Similarity" among the themes.

Yes, all the theme, which they have written there are also transferred, as well as in the "Similarity" button are there links to Movies.

[rick.ca]

I think I've fixed everything mentioned. I appreciate the comments, but I hope you guys appreciate how many brain cells this is costing me...

Version 15 attached (and to top post). Changes:

• changed 'Synopsis' to prevent adding of only a copyright notice                               
• added GET_POSTER option                                                                       
• ...use to disable getting poster for one record if bad image URL is causing HTTP 404 error     
• fixed 'Themes' handling of values behind 'more' button (was duplicating/skipping values)       
• fixed 'Themes' handling of lone theme value (i.e., only button under 'Explore Related Movies')
• changed 'Genres' as recommended by Ivek (works for custom field; mfGenres not tested)         
• converted spaces to tabs for script readability and easier editing                         


[attachment deleted by admin]

[rick.ca]

So I'd have to go in and replace that section every time a new version was released, yes?

I'm not sure what you mean by this. If you're making personal modifications, then keeping it up-to-date can be a PITA. To the extent possible, I recommend doing whatever you can to avoid differences. For example, use the same custom field names as used in the script. (That reminds me—I need to review those to ensure we're using a sensible name in all cases—normally, whatever Rovi calls it.) If there are other essential differences, I recommend using a text editor that does a file comparison. Then you can see the differences and quickly modify one or the other to create an updated personal version.

If you have any modifications you would like to share, we can add them to the script in the same way there are a few RICK_CA options. BTW, I trust no one minds that's 'true' by default.

[RazorHall]

So I'd have to go in and replace that section every time a new version was released, yes?

I'm not sure what you mean by this.

I was referring to Ivek's variation on the code to get those missing themes under the "More" button, a few posts earlier.

[RazorHall]

Really liking Version 15 so far.  

Here's an odd one... I can't get the Rovi Rating to download for the movie Top Model.  I can easily edit the rating myself, but I've always used the "Additional Rating" field, and where every other movie has "AllMovie Rating" overwritten with "Rovi Rating" as the info is downloaded/updated, this one still shows "AllMovie Rating".

EDIT: Found another one, The Dallas Connection, that's behaving as above.

[rick.ca]

Here's an odd one... I can't get the Rovi Rating to download for the movie Top Model.

Although it may seem odd, it does reveal something important. We don't yet have a very good understanding of how the fundamental processing flow of the script is supposed to work. Your Top Model rating is not being updated because nothing is being updated. Following is the ParsePage routine at the end of the script (most of my musings are for my own and Ivek's benefit)...

Code: [Select]

function ParsePage(HTML : String; URL : AnsiString) : Cardinal;
begin
HTML := HTMLToText(HTML);
HTML := StringReplace (HTML, 'http://allrovi.com', 'http://www.allrovi.com', True, True, False);
if Pos('Search Results for', HTML) > 0 then begin
ParseSearchResults(HTML);
Result := prList;
Exit;
end else
if Pos('- Cast, Reviews, Summary, and Awards - AllRovi</title>', HTML) > 0 then
ParseMovie(URL, HTML)
else
if Pos('- Review - AllRovi</title>', HTML) > 0 then
ParseReview(HTML)
else
if Pos('- Cast and Crew - AllRovi</title>', HTML) > 0 then
ParseCast(HTML)
else
if Pos('- Releases - AllRovi</title>', HTML) > 0 then
ParseDVDReleases(HTML);
Mode := NextMode(Mode);
The if-then-else section is testing for possible forms of page title, which presumably indicate what tabs exist on the page. The most common one, "Cast, Reviews, Summary, and Awards," means everything is there. [Edit1: And it seems this is often used even if all the information is not there.] So ParseMovie is called. I don't really understand how, but the other parts somehow get processed in subsequent "Modes." In the case of Top Model, the "Cast and Crew" page title means there's no synopsis or review. So ParseCast is called. It works, but nothing else happens. The if-then-else logic used here is simply wrong and ineffective.

[Edit2:] Sorry. Now I get it. On each change of Mode, a new page (i.e., main, review, cast and crew, releases) is downloaded, and the if-then-else section is testing for which page is loaded. I don't understand why that can't be determined by Mode, but I've tried as see that it can't (or, at least, I couldn't).

Unfortunately, I don't have a clue what to replace it with. If I try an unconditional ParseMovie(URL, HTML), it seems everything is processed three times—once for each of the main page and the two tabs—and the three URLs are saved. My guess is the script doesn't properly follow the original design where each "Mode" is run separately in turn. But it's over my head. If Ivek can't figure it out, we'll need nostra's help (which probably won't come any time soon).

[Edit2:] So the problem is that movies like these examples have main pages with Cast and Crew titles. As a result, ParseCast is done, but ParseMovie is not. Somehow, this situation needs to be detected, and the same page processed by both...

The good news is the proportion of movies affected seems to be very small. Of my 2200 movies, there's only about a dozen for which there's no Cast data. Only six without a Synopsis (one of those is Top Model). Obviously these are rare, and for most of them I wouldn't expect to find much good data anyway. When Rovi's data is incomplete, the quality of what is there often seems questionable. Nevertheless, I would very much like to see the script working properly. It's annoying to have to wonder whether the data retrieved is all there is—or the result of a script error.

[rick.ca]

Sorry for the ranting. Hopefully this solves the issue. But please do look for more odd-ball movies. If there are any sole 'Review' or 'Releases' pages serving as main pages, I'd like to confirm they're properly handled too.

Version 16 attached (and to top post). Changes:

• fixed handing of issue where 'Cast and Crew' serves as the main page
• ...similar fix for sole 'Review' or 'Releases' pages, but haven't found any yet
• changed 'Poster' so .GIF image is no longer downloaded
• ...normally 'no-poster' placeholders

[attachment deleted by admin]

[minolotus]

..., but I hope you guys appreciate how many brain cells this is costing me...

Yes, we do. Thanks again for you effort. The script helps me a lot to keep track of my movies.

And we will keep your lost brain cells in font memory 

A minor issue: I think there is no comment in the script what the rick_ca option includes. The only remark I could find is related to the themes. But looking at the code I assume that this option affects also other thinks? If this is true I would suggest to add a separate comment in the upper part of the script.

[Ivek23]

Sorry for the ranting.

The following code

Code: [Select]

//Title
//modified by rick.ca 07/10/2011
//modified by Ivek23 08/01/2011
curPos := Pos('<title>', HTML) + Length('<title>');
endPos := PosFrom('- Cast, Reviews, Summary, and Awards - AllRovi</title>', HTML, curPos);
TmpStr1 := Copy(HTML, curPos, endPos - curPos)

if TmpStr1 <> '' then

AddFieldValue(mfTitle, TmpStr1);  
AddCustomFieldValueByName('Title', TmpStr1);

if TmpStr1 <> 'title' then
LogMessage('No Title available')
else

curPos := Pos('<title>', HTML) + Length('<title>');
endPos := PosFrom('- Cast and Crew - AllRovi</title>', HTML, curPos);
TmpStr3 := Copy(HTML, curPos, endPos - curPos)

if TmpStr3 <> '' then

AddFieldValue(mfTitle, TmpStr3);  
AddCustomFieldValueByName('Title', TmpStr3);

is added back to the movie title in the custom field and "Rovi rating" again returns to its place in the rating box. With a similar code would also have movie title also recovered in the "Review or Releases".

[rick.ca]

And we will keep your lost brain cells in font memory 

That should be enough for the number that survived. I'm NOT a programmer!

I think there is no comment in the script what the rick_ca option includes.

I assumed you'd search the code for 'RICK_CA' and see what it does, but I suppose this is more humane...

Please note the existence of 'User Options' in the body of the script below. Set those to 'True' or 'False' as explained there. One of those, RICK_CA, is for the personal preferences of rick.ca, where they differ from those of Ivek23. Those are...

      •   add footers in the form, "—Author" to ~mfDescription~ and ~Review~
      ...used instead of headers.
      
      •   save just a link to 'Releases' page in ~mfFeatures~
      ...used instead of saving entire page to custom field;
      ...as this still requires use of links to view specific DVD information.
      
      •   display LogMessage that these options are being used.

But when I look at this, it seems rather silly. If Ivek would just concede my Description and Review are much prettier than his, we could do away with this confusing option. If the Releases link is not wanted, mfFeatures can be disabled in  Overwrite fields..., so it's not necessary for that. What do you say, Ivek?

[rick.ca]

The following code...is added back to the movie title in the custom field and "Rovi rating" again returns to its place in the rating box.

Does this work? It seems to, but I don't use either field...

Code: [Select]

//~mfTitle~ or ~Title~

curPos := Pos('<title>', HTML) + Length('<title>');

if Pos('- Cast, Reviews, Summary, and Awards - AllRovi</title>', HTML) > 0 then
endPos := PosFrom('- Cast, Reviews, Summary, and Awards - AllRovi</title>', HTML, curPos)
else
endPos := PosFrom('- Cast and Crew - AllRovi</title>', HTML, curPos);
TmpStr1 := Copy(HTML, curPos, endPos - curPos);

AddFieldValue(mfTitle, TmpStr1); 
AddCustomFieldValueByName('Title', TmpStr1);


//Save start position for Movie data
topPos := PosFrom('<div class="page-heading">', HTML, curPos)


//~mfYear~ or ~Year~

endPos := PosFrom('</span></h1>', HTML, curPos);
curPos := endPos - 5;
TmpStr2 := IntToStr(StrToInt(Copy(HTML, curPos, 4)));

if TmpStr2 = '0' then
TmpStr2 := '';

AddFieldValue(mfYear, TmpStr2);
AddCustomFieldValueByName('Year', TmpStr2);

if TmpStr2 = '' then
TmpStr2 := 'year unknown';

if RICK_CA then
LogMessage('Parsing ' + TmpStr1 + '(' + TmpStr2 + ') using rick.ca script options')
else
LogMessage('Parsing ' + TmpStr1 + '(' + TmpStr2 + ')');

I'm also curious why you need to save a custom Title. Is that because you use mfTitle for titles translated to you native language?

With a similar code would also have movie title also recovered in the "Review or Releases".

Sorry, I don't understand. Do we need to make further changes?