AllRovi movie script

[rick.ca]

Notice:

AllRovi script does not work anymore.

It's topic not very visible on the Scripts and Templates board, so I'm announcing it here. Thanks to Ivek, we once again have access to most Allmovie data.

For those not familiar with Rovi/Allmovie, its consistent quality of Synopsis and Reviews is much better than the user-provided crap on IMDb.

Now for all former AllMovie fans here is new AllRovi script, based on AllMovie script and with the help of other scripts, especially the new Italian script. The script gets most of the movie information currently available at the new AllRovi site, including:

• Basic info; Title, Year, Director, Genre, Run time, Release, etc.
• Poster (not good quality, but sufficient for a thumbnail)
• Rovi rating
• Synopsis
• Review
• Sub-Genres and Themes
• MPAA rating and Flags

We (co-authors Ivek23 and rick.ca) have a faint hope more data that was available at the old AllMovie site will be added to the new AllRovi site. If that happens, we'll update the script to include it.

The most current version of the script is always attached to this message. Details of changes are documented in the script, and may be commented on in messages posted to this topic. Please post any comments or questions concerning this script to this topic.

[rick.ca]

The script is now very near completion. A "Release Candidate," if you like. I've added a list of it's unique features to it's description...

...and adds some special features:

• Full Cast and Crew listings with links to Rovi bios.
• Support for separate Attributes, Themes and Moods, even though Rovi now lumps all these together in Characteristics.
• Mood links to old AllMovie mood pages (description of the mood and a list of "major works") found in the Web Archive.
• Links to Rovi's Genre and Sub-Genre pages (with description, significant works and 'most visited' films).

[rick.ca]

The script is now very near completion. A "Release Candidate," if you like.

Who an I kidding? This is not Microsoft. The script is complete and has been released to the auto-update system. If there you have any problems or suggestions for improvement, please post them here.

[minolotus]

Even the script is now officially released, I would like to add two comments on the latest version.

I think the new update field is a really good idea (it would be nice if a similar field for the imdb script exists ... but that’s another point). Unfortunately, the date format is not similar to the date format of the "added" and "modified" fields. Would it be possible that the fields use the local settings of the respective computer? 
 
I have another issue with the "modified" field. I am not sure if this is related to the script or a more general problem of PVD. I rerun the script to get the synopsis to a custom field. The script put the data to the field but the "modified" field wasn’t updated.  I checked this with the "characteristics" and the "flags" field (removing the data and rerunning the script). In both cases the "modified" field wasn’t updated?

[Ivek23]

I think the new update field is a really good idea (it would be nice if a similar field for the imdb script exists ... but that’s another point). Unfortunately, the date format is not similar to the date format of the "added" and "modified" fields.  

"Added" field does not change, remaining from the date of entry when the entry was added, while in the "Modified" field is changing the date in it with every change made ​​to any of the saved in PVD database.

I have another issue with the "modified" field. I am not sure if this is related to the script or a more general problem of PVD.

I think, this is more

a more general problem of PVD

What about the custom "Updated" field, there are certain data change.
See attached pictures.


[attachment deleted by admin]

[minolotus]

"Added" field does not change, remaining from the date of entry when the entry was added, while in the "Modified" field is changing the date in it with every change made to any of the saved in PVD database.

My point was that the format of the date of the "update" field on the one hand and the "modified" and "added" fields on the other hand are not the same due to the local date settings on my computer. The structure of the update field is "YYYY-MM-DD" and the structure of the other fields mentioned are "DD.MM.YYYY".

What about the custom "Updated" field, there are certain data change.
See attached pictures.

That's true. The "update" field works fine. But it is important for me that the "modified" filed is updated every time when changes on the data are proceeded.

[rick.ca]

Unfortunately, the date format is not similar to the date format of the "added" and "modified" fields. Would it be possible that the fields use the local settings of the respective computer?

The script engine includes the necessary date-handling functions, but it's not possible to add a custom date-type field to version 0.9. It is in version 1.0, so we can make that change when it's released.

I have another issue with the "modified" field. I am not sure if this is related to the script or a more general problem of PVD. I rerun the script to get the synopsis to a custom field. The script put the data to the field but the "modified" field wasn’t updated.  I checked this with the "characteristics" and the "flags" field (removing the data and rerunning the script). In both cases the "modified" field wasn’t updated?

This is curious. If I change any fields (e.g., removing data to see what happens when the script is rerun), Modified is updated (i.e., immediately—before the script is run). I don't know why that might be different for you. In any case, the main program handles the date fields. Whatever is happening has nothing to do with the script.

But it is important for me that the "modified" filed is updated every time when changes on the data are proceeded.

As well you should be, if using PvdImport to update J. River Media Center. It relies on an accurate DateTime to determine which records have been updated by PVD. I haven't noticed any problems with records failing to update. Have you?

[minolotus]

The script engine includes the necessary date-handling functions, but it's not possible to add a custom date-type field to version 0.9. It is in version 1.0, so we can make that change when it's released.

Thanks, that's great. Looking forward to it.

This is curious. If I change any fields (e.g., removing data to see what happens when the script is rerun), Modified is updated (i.e., immediately—before the script is run). I don't know why that might be different for you. In any case, the main program handles the date fields. Whatever is happening has nothing to do with the script.

Are you sure that the update of the field is caused by the rerun of the script and not by removing the data before? I check this with the imdb script and I think that the timestamp is set after the update?

As well you should be, if using PvdImport to update J. River Media Center. It relies on an accurate DateTime to determine which records have been updated by PVD. I haven't noticed any problems with records failing to update. Have you?

Yep, when the "modified" field is not updated, PvDImport does not recognise the update and doen't import the new data.   Then, I need to start the update manually

Update: I have just remembered that I had similar problems with the multiple movies editor:

I have discovered that the field “last modified” is not always updated when using the multiple movies editor. This appears only from time to time. Maybe the problem is related to the field to be modified. When I change the genre, the timestamp does not change but after editing the director field of the same movies the “last modified” field is updated.

[rick.ca]

Are you sure that the update of the field is caused by the rerun of the script and not by removing the data before? I check this with the imdb script and I think that the timestamp is set after the update?

Modified should be updated whenever any data changes for any reason. I've never had reason to care, but it actually improperly updates if a record is put in edit mode and no changes are made.

Although I haven't tested all fields, I haven't noticed or found any that don't cause Modified to be updated when they are changed by directly editing the record. Have you? Which fields?

Although I haven't tested all plugins and scripts, I've never noticed Modified not being updated when one is run.

I have just remembered that I had similar problems with the multiple movies editor:

I can reproduce this. I haven't attempted to determine whether fields other than Genre are affected, or whether or not it happens consistently. From what I'm seeing, my guess is the problem is restricted to the MME.

Yep, when the "modified" field is not updated, PvDImport does not recognise the update and doen't import the new data.

This observation doesn't shed any light on the matter. If Modified has not been updated by PVD, PvdImport is not supposed to re-import the data.

Then, I need to start the update manually

What do you mean by this? If Modified hasn't changed, you would have to force an update by using the "Send to" menu or run another "Mass import."

[minolotus]

I did some tests to narrow down the source of the problem but this wasn’t really successful. I couldn’t discover a pattern in the behaviour of the “modified” field. Here are some of my observations:

-   The problem is not related to a specific script or to the import by scripts in general. The problem could occur with the imdb script and allrovi script, the MME and also in the manual edit mode.
-   I tested also with a new and clean database therefore this should not be problem with a corrupted database. Further I switched between local access to the database and connection to the firebird server but this did have any affects on the problem.
-   My movie skin could be blamed as I tried this also with a standard skin of PVD.
-   There are some fields which causes never any problem, i.e. Budget, box office. When updating the data with a script, deleting or editing the entries manually the “modified” field is updated.


The problem seems to be connected with specific movies as well as with specific fields. Please find attached a table with my testing for four movies. There are some actions which causes always problems (update sub-genre and characteristics with allrovi script). Some other problems occur only for movie 2 and 4.  


[attachment deleted by admin]

[minolotus]

Then, I need to start the update manually

What do you mean by this? If Modified hasn't changed, you would have to force an update by using the "Send to" menu or run another "Mass import."

That's what I meant. 

[minolotus]

-   The problem is not related to a specific script or to the import by scripts in general. The problem could occur with the imdb script and allrovi script, the MME and also in the manual edit mode.

One additional remark regarding the imdb script: In the table in my post above the update with the imdb script always results in an update of the "modified" field. But there are also examples when this is not working: When I remove itemes from the "producer" field. the "modified" filed is always updated. But when I rerun the imdb script the "producer" field is filled properly but the "mofified" filed leaves untouched. I checked this on several movies and got on all of them the same problem.

[rick.ca]

Have you included the custom [Updated] field in your database? I have, and it, of course, always changes when the AllRovi script is run. This, in turn, causes [moddate] to be updated. I have no idea why it's not updating as expected, but if running either the plugin or the script will always update it, where does a problem remain? Are there any fields not updated from external sources that do not cause [moddate] to be updated when changed?

The issue I'm seeing, BTW, seems to be sort of the opposite problem. My [moddate] seems to be updated just by putting a record into edit mode—without changing anything. This means, of course, I'm not able to replicate anything you're reporting.

[minolotus]

Have you included the custom [Updated] field in your database?

Yes, I have. And this field never causes any problems. It is always updated when running the allrovi script.

Are there any fields not updated from external sources that do not cause [moddate] to be updated when changed?

I am sorry. Could you please restate your question for me? I have a problem to understand what you mean.

The issue I'm seeing, BTW, seems to be sort of the opposite problem. My [moddate] seems to be updated just by putting a record into edit mode—without changing anything.

Yes, I discovered this, too, but only for some movies and not for all. At the "Living Daylights, The" for example, the [moddate] field is not updated when entering the edit mode.  

I am stumped  

Update: Now, I am getting officially crazy. This morning, when removing the entries from the sub-genre field for “licence to kill” in the edit mode, the [moddate] field was properly updated. But after rerunning the allrovi script, the [moddate] was not updated. I repeated these steps several times and checked carefully the [moddate] field.  So, I am sure that this wasn’t an error in observation.
 
Now, I did this test again and now it is working properly. The [moddate] field is updated in both cases: after removing the content from the field in the edit mode and after running the script.

After this I created a new database and added as custom fields [updated] and [sub-genre]. I added the “licence to kill” movie and ran the imdb and allrovi script. Then, I removed the entries from the sub-genre field and closed the edit mode… The [moddate] field hasn’t been updated  The update with the allrovi script worked fine including an update of the [moddate]  field. I repeated the three times and results were always the same.

I don’t see any pattern in this. I think I will throw my computer away and move into the forests.

[RazorHall]

I'm seeing a few movies that aren't getting the Characteristics info, such as Phantom Love.

[Ivek23]

I'm seeing a few movies that aren't getting the Characteristics info, such as Phantom Love.

Same cases a major part in Bounty Huntress 2 (2002).
Synopsis will be postponed until the end ParseMovie As mentioned here, follows attempts:

this code

Code: [Select]

//'Synopsis' saved to ~mfDescription~

curPos := PosFrom('<span>', HTML, curPos);
endPos := PosFrom('</span>', HTML, curPos);
TmpStr5 := TextBetween(HTML, '<span>', '</span>', True, curPos);

curPos := PosFrom('<span>by', HTML, curPos);
endPos := PosFrom('</span>', HTML, curPos);
TmpStr3 := TextBetween(HTML, '<span>by ', '</span>', True, curPos)

curPos := PosFrom('<strong>', HTML, curPos);
endPos := PosFrom('</strong>', HTML, curPos);
TmpStr4 := TextBetween(HTML, '<strong>', '</strong>', True, curPos);

curPos := PosFrom('<p>', HTML, curPos);
endPos := PosFrom('</p>', HTML, curPos);
TmpStr := Copy(HTML, curPos, endPos - curPos);
TmpStr1 := StringReplace(TmpStr, ' -- ', '—', True, True, False);
TmpStr2 := StringReplace(TmpStr1, ' --- ', '—', True, True, False);
TmpStr := StringReplace(TmpStr2, '--', '—', True, True, False);
TmpStr1 := StringReplace(TmpStr, #13#13#13#13, #13#10#13#10, True, True, False);
TmpStr2 := RemoveTagsEx(TmpStr1);

if (TmpStr4 = 'synopsis') AND (Pos('©', TmpStr2) = 0) then
AddFieldValue(mfDescription, TmpStr2 + #13 + '—' + TmpStr3)
else
if (TmpStr4 <> 'synopsis') OR (Pos('©', TmpStr2) = 1) then
LogMessage(' Synopsis: Not available');

Copy here as shown in the attached code

Code: [Select]

//~AMG ID~

TmpStr := HTMLValues(HTML,
'<dt>AMG ID</dt>', '</dd>',
'<pre>', '</pre>',
', ', endPos);
AddCustomFieldValueByName('AMG ID', TmpStr);


//'Synopsis' saved to ~mfDescription~

curPos := PosFrom('<span>', HTML, curPos);
endPos := PosFrom('</span>', HTML, curPos);
TmpStr5 := TextBetween(HTML, '<span>', '</span>', True, curPos);

curPos := PosFrom('<span>by', HTML, curPos);
endPos := PosFrom('</span>', HTML, curPos);
TmpStr3 := TextBetween(HTML, '<span>by ', '</span>', True, curPos)

curPos := PosFrom('<strong>', HTML, curPos);
endPos := PosFrom('</strong>', HTML, curPos);
TmpStr4 := TextBetween(HTML, '<strong>', '</strong>', True, curPos);

curPos := PosFrom('<p>', HTML, curPos);
endPos := PosFrom('</p>', HTML, curPos);
TmpStr := Copy(HTML, curPos, endPos - curPos);
TmpStr1 := StringReplace(TmpStr, ' -- ', '—', True, True, False);
TmpStr2 := StringReplace(TmpStr1, ' --- ', '—', True, True, False);
TmpStr := StringReplace(TmpStr2, '--', '—', True, True, False);
TmpStr1 := StringReplace(TmpStr, #13#13#13#13, #13#10#13#10, True, True, False);
TmpStr2 := RemoveTagsEx(TmpStr1);

if (TmpStr4 = 'synopsis') AND (Pos('©', TmpStr2) = 0) then
AddFieldValue(mfDescription, TmpStr2 + #13 + '—' + TmpStr3)
else
if (TmpStr4 <> 'synopsis') OR (Pos('©', TmpStr2) = 1) then
LogMessage(' Synopsis: Not available');

end;


procedure ParseSearchResults(HTML : String);

and this for me this works perfectly, how it works for you now.

[rick.ca]

and this for me this works perfectly, how it works for you now.

Thanks, Ivek. Yes, that seems to work fine. It would be nice to understand why, but that's okay.

I'll upload version 1.1 now. The change is referenced to your message above, so there's no need to post a script or explanation anywhere else.

[Ivek23]

I'll upload version 1.1 now. The change is referenced to your message above, so there's no need to post a script or explanation anywhere else.

OK, I understand.

[deazo]

 Hi Ivek23 and Rick, I already thanked you for this but hey thank you again!
 
 I have a slight problem. For some reason, Allrovi script does not gather the rating anymore. I have not changed anything to the preferences, the "additional rating" box is checked. I wonder if there was a change in the last version that caused this?

[Ivek23]

When updating data with AllRovi script for the movie Dakota (1945), I could not find this title in the results table, where several movie with the same title and a different year.
The problem is procedure ParseSearchResults in this part of code:

Code: [Select]


...

//if there are more movies with the same or similar title...

curPos := Pos('<div class="results">', HTML);
if curPos < 1 then
Exit;

curPos := PosFrom('<a href="http://www.allrovi.com/movies/', HTML, curPos);
while curPos > 0 do begin
endPos := PosFrom('">', HTML, curPos);
URL := Copy(HTML, curPos + 31, endPos - curPos - 31);
URL := BASE_URL + URL;

curPos := PosFrom('">', HTML, curPos);
endPos := PosFrom('</a>', HTML, curPos);
Title := Copy(HTML, curPos + 2, endPos - curPos - 2);

curPos := PosFrom('<td class="year">', HTML, curPos);
endPos := PosFrom('</td>', HTML, curPos);
Year := Trim(Copy(HTML, curPos + 17, endPos - curPos - 17));

AddSearchResult(Title, '', Year, URL, '');
curPos := PosFrom('" href="/movies/', HTML, curPos);
end;

end;

function NextMode(curMode : Integer) : Integer;


When I am this one snippet of code  curPos := PosFrom('" href="/movies/', HTML, curPos); replaced by tiny fraction of the code curPos := PosFrom('<a href="http://www.allrovi.com/movies/', HTML, curPos);, now in the results table also previously found above the movie title.

Now this piece of code looks like this:

Code: [Select]


...

//if there are more movies with the same or similar title...

curPos := Pos('<div class="results">', HTML);
if curPos < 1 then
Exit;

curPos := PosFrom('<a href="http://www.allrovi.com/movies/', HTML, curPos);
while curPos > 0 do begin
endPos := PosFrom('">', HTML, curPos);
URL := Copy(HTML, curPos + 31, endPos - curPos - 31);
URL := BASE_URL + URL;

curPos := PosFrom('">', HTML, curPos);
endPos := PosFrom('</a>', HTML, curPos);
Title := Copy(HTML, curPos + 2, endPos - curPos - 2);

curPos := PosFrom('<td class="year">', HTML, curPos);
endPos := PosFrom('</td>', HTML, curPos);
Year := Trim(Copy(HTML, curPos + 17, endPos - curPos - 17));

AddSearchResult(Title, '', Year, URL, '');
curPos := PosFrom('<a href="http://www.allrovi.com/movies/', HTML, curPos);
end;

end;

function NextMode(curMode : Integer) : Integer;


As well as images attached as evidence of how the before and after correcting codes.




[attachment deleted by admin]