Personal Video Database 1.0.2.7 MOD

[VVV_Easy_Programing]

I have found different url formats versions in my PVD too?
Can we write a fast url standards?
My proposal is (remember that we cann't use httpS):
IMDB Movie (finish with '/'=Slash):

• Movie=http://www.imdb.com/title/ttxxxxxxx/
• Person=http://www.imdb.com/name/nmxxxxxxx/

TMDB (NO finish with '/'=Slash):

• Movie=http://www.themoviedb.org/movie/xxxxxxx
• Person=http://www.themoviedb.org/person/xxxxxxx

FilmAffinity (finish with '.html', maybe spanish (several countries) or english):

• Movie Spanish= http://www.filmaffinity.com/es/filmxxxxxxx.html
• Movie English= http://www.filmaffinity.com/en/filmxxxxxxx.html

BTW, in the last times of imdb.dll pluging it retrived bad movie and people url (for instance for directors with URL+'?ref_=tt_ov_dr' and writers URL+'?ref_=tt_ov_wr'). In that moment I wrote a little SQL script (you can use it in debug PVD mode in one hidden option of Help menu) in order to correct the database:
                 update PEOPLE set "url"=replace("url",'?ref_=tt_ov_dr', '/');
                 update PEOPLE set "url"=replace("url",'?ref_=tt_ov_wr', '/');
perhaps it may help to correct something.

I just wanted to ask if it's possible FA to have it's own rating custom field?

In the next MOD version I will include the solution that Ivek was advancing
 AddCustomFieldValueByName('FilmAffinity_Rating',ItemValue);
because it doesn't nothing bad for the "normal" users but you need add by yourself this Custom field.

BTW, I have included several 'hidden' Custom Fields of Ivek23 in the scripts.
Ivek23, perhaps can be useful for other users open a new Thread with the information "Possibles improving Custom Fields" working in MOD version, how is the information and how add in the PVD database.

[afrocuban]

Thank you for your feedback, guys. Good to know that I'm not the one who messed up database, since you two have also people duplicates. Thanks for FA rating and all "hidden" custom fields.
I tried your SQL script in debug mode, cause I also notice mistakes in imported urls. Thank you for that. It didn't want to run with "As script" unchecked (some dynamic error...) so I ran it "As script", and although my database is which is over 15GB, with posters it finished in 3-4 minutes claiming "Script successfully executed"
After that I tried to update movies I stated above, but no luck. But, for example, I just repeated update for "All About Eve" which didn't succeed at first, and now it succeeded?! It never happened in original PVD, and the only conclusion I can make is that if some field isn't rewritable (for God knows what reason) the record cannot be updated.

I'd appreciate any further help on this and am more than grateful for your tremendous work and efforts to make PVD alive!

My best regards to you two.

[VVV_Easy_Programing]

Have you used the Tools/Optimize database ... ?
Perhaps it solved your "write protected" records but I warm you that it spend a lot, a lot of time.
With a 7.000 movies you must have more of 200.000 persons!
Perhaps you will spent one or two days with PVD working and the PC un-responding.

[afrocuban]

Thanks for your idea, VVV. I remember I used that option at the very beginning, and that then as I can remember it deleted some records I didn't want to. Do you by any chance know what exactly optimizing will do to my database? I hae never ever had some problem with PVD database speed, but whatever - I'll do it if my records will not be deleted or database corrupted.

Kind regards

[afrocuban]

Dear VVV and Ivek,

Meanwhile, I have experienced another strange issue. While updating some movies, it asks me to confirm the movie, although IMDb urls are already in database, and for for twice!. You can check comparing offered url on the first screenshot (question marks), with the url already in database in the second screenshot (red arrow) - they are the same.

Also, for some movies I get infobox "Movie not found " although IMDb url exists - please see third screenshot?!

It didn't happen earlier.

It's like script doesn't use existing urls. Why is that?

Any idea? Thank you in advance.

[VVV_Easy_Programing]

This is easy: in both "13" and "After life"  imdb url's lacks "www" remeber the standar url is:
http://www.imdb.com/title/ttxxxxxxx/

Perhaps you can use the SQL script:

update MOVIES set "url"=replace("url",'http://imdb', 'http://www.imdb');
.

BTW, one tools that can help you is pass the db to Excel with the export template "CSVDB" and search with the excel filters some bizarre things.

Do you by any chance know what exactly optimizing will do to my database?

It will do as you select: delete duplicates entries, delete orphan persons (persons not related with your movies). You can select if it will do with all movies/people or only qith the visible (so you can protect movies with a filter and hidde it's)

[afrocuban]

Thanks VVV. It's clear now.

Is it possible then, considering your statement:

I have found different url formats versions in my PVD too?
Can we write a fast url standards?
My proposal is (remember that we cann't use httpS):
IMDB Movie (finish with '/'=Slash):

• Movie=http://www.imdb.com/title/ttxxxxxxx/
• Person=http://www.imdb.com/name/nmxxxxxxx/

TMDB (NO finish with '/'=Slash):

• Movie=http://www.themoviedb.org/movie/xxxxxxx
• Person=http://www.themoviedb.org/person/xxxxxxx

FilmAffinity (finish with '.html', maybe spanish (several countries) or english):

• Movie Spanish= http://www.filmaffinity.com/es/filmxxxxxxx.html
• Movie English= http://www.filmaffinity.com/en/filmxxxxxxx.html

... to standardize urls in form:

[*.]imdb.com/title/ttxxxxxxx/
[*.]imdb.com/name/nmxxxxxxx/

[Ivek23]

Dear VVV and Ivek,

Meanwhile, I have experienced another strange issue. While updating some movies, it asks me to confirm the movie, although IMDb urls are already in database, and for for twice!. You can check comparing offered url on the first screenshot (question marks), with the url already in database in the second screenshot (red arrow) - they are the same.

Also, for some movies I get infobox "Movie not found " although IMDb url exists - please see third screenshot?!

It didn't happen earlier.

It's like script doesn't use existing urls. Why is that?

Any idea? Thank you in advance.

I myself have written here why this happens, and who is responsible for problems with IMDB url addresses. This is the IMDB plugin, which caused problems to the afrocuban and also to me, but I edited it manually by adding " www " letters in the IMDB url.

There is a possibility that duplicates are people for certain reasons, such as different url addresses (or they are not at all) for people. I have already noticed this in my IMDB records for people. For  the IMDB plugin was the download URL of the Imdb reference pages without " www " in the url address. IMDB script has downloaded a full url with Main (overview) Imdb pages.

Finally, I had a similar problem with the TheMovieDB [API] script, where there was a problem with url titles for the Director because the script code was incorrectly written in the script and there were duplicate names for the director because of this, but not for all, only in certain cases.

[VVV_Easy_Programing]

Is it possible then, considering your statement:
... to standardize urls in form:
[*.]imdb.com/title/ttxxxxxxx/
[*.]imdb.com/name/nmxxxxxxx/

I think to do that this is dificult because the script use the parameter
BASE_URL_PRE       = 'http://www.imdb.com/title/''
to make the url and for search if exist the url.

And this is risky because if the original PVD came to live again it use the same behavior with the Function GetBaseURL for locate the url.

But Ivek23 and you don't need add 'www' manual the SQL script (remeber, in debug mode Help/SQL, As script checked) 
update MOVIES set "url"=replace("url",'http://imdb', 'http://www.imdb');
works fine: I test it.

[afrocuban]

Hello VVV,

Thank you for another SQL tip. I applied it immediately upon you first right about it, and it worked like a charm! I just thought if it's possible to generalize the term as much as possible, because we don't know what will happen with IMDb code in the future and with maintain of PVD and IMDb script, but hey - we will survive. I have my database for 20 years, from the first days of "WhereIsIt?" program, til final solution - PVD.

About optimizing database, thank you for explanation, but I think I mustn't use it because it looks it will do exactly what I don't want at any cost - it'll delete duplicate people, so I would loose them in movies, so I would again have to update 7000 movies to get people back, again most probably to get duplicates. The only solution to duplicates that I see is to delete all people, then to update all movies with people records (director, actor, producer, writer, composer, etc...).

Maybe I'd do that, but only when I'm sure that I will be able to update movies without breaks that I have now.

To add to this, I hve experienced one more IMDb script issue:
- If URL doesn't have slash - "/" at the and, we get "404" error message, like URL doesn't exist on imdb.com, while actually url is valid. Please see first screenshot attached.
- After pressing OK (1-5 times, depending on how many pages are imported), script adds ghost link: - "http:/"? Please check second screenshot.

Anything we can do about this?

Best regards

[Ivek23]

Now I've tried to update The Villain movie title, where there is already information and has such an url address.

http://imdb.com/title/tt0080097/

The interesting thing is that I did not have any problems in updating the information.

[Ivek23]

BTW, I have included several 'hidden' Custom Fields of Ivek23 in the scripts.
Ivek23, perhaps can be useful for other users open a new Thread with the information "Possibles improving Custom Fields" working in MOD version, how is the information and how add in the PVD database.

To that, when I do some more tests, because I discovered some more errors and I still test some more improvements to the code sections for IMDB_ [EN] [HTTPS] script.

[VVV_Easy_Programing]

The interesting thing is that I did not have any problems in updating the information.

Is sure that the script don't find the IMDB url. Maybe that the script are always searching the movie URL's in IMDB page but it find at first and don't ask nothing. You can see that with the Log tool. In this movies, after pass the script, don't you have two IMDB url's stored?

To add to this, I hve experienced one more IMDb script issue:
- If URL doesn't have slash - "/" at the and, we get "404" error message, like URL doesn't exist on imdb.com, while actually url is valid. Please see first screenshot attached.
- After pressing OK (1-5 times, depending on how many pages are imported), script adds ghost link: - "http:/"? Please check second screenshot.

This is not easy to solve with a SQL script so I code a not critical MOD patch for you. In the next post.

[VVV_Easy_Programing]

Attached non critical Acumulative Scripts Patch date:2018.10.07 (to copy in PVD folder for overwriting the existing script files).

Changelog:
IMDB_[EN][HTTPS].psf and IMDB_People_[EN][HTTPS].psf:

 V 1.2.0.2 (07/10/2018) Managed IMDB url's not finish in BASE-URL-SUF='/' and principal page download errors.

[Ivek23]

The interesting thing is that I did not have any problems in updating the information.

Is sure that the script don't find the IMDB url. Maybe that the script are always searching the movie URL's in IMDB page but it find at first and don't ask nothing. You can see that with the Log tool. In this movies, after pass the script, don't you have two IMDB url's stored?

I'm sure and I stand for it, because the next url did not cause any problems when I updated the information. I checked this several times already due to the afrocuban major problems described in this regard.

There Was A Crooked Man...       http://imdb.com/title/tt0066448/

No, I do not have any duplicate URLs.

[VVV_Easy_Programing]

I'm agree with you that the script doesn't give you any problems.
I want to say that perhaps this is because the script don't use the stored URL and it search the title in IMDB and continues scrapping.
You may see the script behaviour with the log window tool.

[afrocuban]

Thank you so much for your help guys! I will try cumulative script patch after finish updating.
Regardless of Ivek's and mine issue regarding "www", it is obviously solvable, one way or another. Still, the original, first issue is mystery and I connected it with the issue that I cannot overwrite some fields (mostly "Title", but some other as well), manually or automatically. Do you have any idea why some field could not be editied, manually, or automatically???

Thanks in advance!

[afrocuban]

I have found the cause of the error from this message:
http://www.videodb.info/forum_en/index.php/topic,4134.msg20748.html#msg20748

... and the workaround for it.

The problem is IMDb url that is stored in the "URL" field! Although it is perfectly valid (I copy it from the URL field then paste it in browser and it goes to the proper movie), it will not work if you try to update movie in PVD, or to manually edit any other field imported by IMDb script or plugin!

So, I went to Grid View, and for some movies I tried to delete Title and Origititle and, bam - I got the same error message as the one while importing with script.

Finally, in Grid view I deleted stored IDMb url (which is perfectly valid), and then let the script imports data and, another BAM - everything worked great!

Meaning, when I delete stored IMDb url (it is sometimes possible in Tree View, but it is always possible in Grid View), I am able to edit any other field, hence to update movie!

Why is this happening and why to only certain movies/records it's the mystery probably no one will solve.

If someone can explain this I'd be more than grateful. Luckily, I had only 105 movies of 7000 movies with this issue, so I will manually edit each one url and update it, but that will not resolve possible future issues. Workaround is not a solution, that is why it is called only - workaround.

Best regards

[VVV_Easy_Programing]

Good news even if it must expend "some" manual editing time! It's true that the Grid View edition is a little bit forgotten.
Perhaps you have some especial chars in the URL's fields that the browser is capable to manage and PVD crash (for instance, the URL's must be separated by spaces not by Return, char#13).

Regardless of Ivek's and mine issue regarding "www", it is obviously solvable, one way or another.

I strongly recommend you use the standar url formats because there is other script (TheMovieDB) that use the IMDB link and is the normal way of PVD to detect the provider url (remember when PVD directly download the page when it detects the BASE_URL in the URL field, this is the reason that in the HTTPS scripts the function GetBaseURL returns BASE_URL_ROUNDABOUT in order to cheat GET PVdB funtion and rondabout the "https" fail).

And if all records are with those formats are easy to change with a SQL script because the URL field is a little bit especial, for instance, I haven't found the manner to protect it from script write or to delete all with the mass editor.

[Ivek23]

IMDB_ [EN] [HTTPS] script

A beauty correction for a piece of code for Date ~ Updated ~ (choose simple or verbose version) .

    //Date ~Updated~ (choose simple or verbose version)
        Date:=DateToStr(CurrentDateTime);
        //AddFieldValueXML('viewdate',Date); //Only date, don't admit time-. Set Seen value at the same time.
        //AddFieldValueXML('moddate',Date+' '+TimeToStr(CurrentDateTime)); //Block the dB saving   
        ExplodeString(Date,DateParts,'-');
        Date:=DateParts[2]+'.'+ DateParts[1]+'.'+DateParts[0]; 
        Date:=StringReplace(Date,'01.','1.',True,True,False);
      Date:=StringReplace(Date,'02.','2.',True,True,False);
      Date:=StringReplace(Date,'03.','3.',True,True,False);
      Date:=StringReplace(Date,'04.','4.',True,True,False);
      Date:=StringReplace(Date,'05.','5.',True,True,False);
      Date:=StringReplace(Date,'06.','6.',True,True,False);
      Date:=StringReplace(Date,'07.','7.',True,True,False);
      Date:=StringReplace(Date,'08.','8.',True,True,False);
      Date:=StringReplace(Date,'09.','9.',True,True,False);         
        //AddCustomFieldValueByName('Updated',Date); // Simple
        //AddCustomFieldValueByName('Updated',Date+' at '+TimeToStr(CurrentDateTime)); // Verbose
        AddCustomFieldValueByName('Updated',Date+' at '+TimeToStr(CurrentDateTime)+' • '+SCRIPT_NAME+' '+SCRIPT_VERSION);  // Annoying

For Function ParsePage_IMDBMovieMPAA is also a small correction again.

Function ParsePage_IMDBMovieMPAA(HTML:String):Cardinal; //BlockOpen
    //Returns:
    //     Result:=prFinished; Script has finished gathering data
    //     Result:=prError;  If any big problem? with exit
    //Retrieve: ~MPAA~
  Var
    curPos:Integer;
    mpaaValue:String;
  Begin
.
.
.
    mpaaValue:=StringReplace(mpaaValue,'Suggested Rating','. Suggested Rating',False,False,False); //Avoid union in films with 'Suggested Rating'
   mpaaValue:=StringReplace(mpaaValue,'update: ','',False,False,True);                         //Cleanig value. //WEB_SPECIFIC
   mpaaValue:=StringReplace(mpaaValue,': ','',False,False,True);                         //Cleanig value. //WEB_SPECIFIC
    curPos:=Pos('Edit',mpaaValue);                                     //WEB_SPECIFIC.
    If 0<curPos then mpaaValue:=Copy(mpaaValue,0,curPos-1);   
    If (mpaaValue='Approved') Then mpaaValue:='Rated G';
.
.
.
    AddFieldValueXML('mpaa',mpaaValue);
    LogMessage('      Get result MPAA='+mpaaValue+'||');
    LogMessage('Function ParsePage_IMDBMovieMPAA END=====================||');
  End; //BlockClose

Full Cast and Crew section

For directors is also a small correction again.

    //Go to "Directed by" ~crew~ctDirectors
    curPos:=Pos('<h4 class="dataHeaderWithBorder">Directed by',HTML);                     //Strings start which opens the block content data. WEB_SPECIFIC   
    If 0<curPos Then Begin
      curPos:=PosFrom('</h4>',HTML,curPos);                                              //Strings end which opens the block content data.  WEB_SPECIFIC
      curPos:=curPos+Length('</h4>');                                                    //Strings end which opens the block content data.  WEB_SPECIFIC
      //Get all "raw" crew summary (in raw because we need the hidden person links) May one person or severals in the ~crew~ n the ~crew~
      endPos:=PosFrom('</table>',HTML,curPos);                                             //Strings which opens/closes the data. WEB_SPECIFIC
      ItemList:=Copy(HTML,curPos,endPos-curPos);
      //LogMessage('           Parse results ('+IntToStr(curPos)+','+IntToStr(endPos)+') complex ItemList:'+ItemList+'||');
      curPos:=Pos('<td class="name">',ItemList)                                         //String which opens the subList data. WEB_SPECIFIC
      index:=1;
      While curPos>0 Do Begin
         If (index>PEOPLE_LIMIT) Then break;     //Limited depassed (Remember index begin in 0).
         //Get PersonURL (Always must it has)
         PersonURL:=BASE_URL_PERSON_PRE + TextBetWeen(ItemList,'<a href="/name/','/',false,curPos) + BASE_URL_SUF;     //Strings which opens/closes the data. WEB_SPECIFIC
         LogMessage('      Parse Results PersonURL:'+PersonURL+'||');         
         //Get Name (Always must it has)
         Name:=TextBetWeen(ItemList,'> ','</a>',false,curPos);                                 //Strings which opens/closes the data. WEB_SPECIFIC     
         LogMessage('      Parse Results Name:'+Name+'||');
         //Get Role: PVD don't save Role in crew people but is added for Ivek23 as LogMessage compatibility
         //Role:=TextBetWeen(ItemList,'<td class="credit">','</td>',false,curPos);                    //Strings which opens/closes the data. WEB_SPECIFIC
            Role:=TextBetWeen(ItemList,'<td ','</td>',false,curPos);                    //Strings which opens/closes the data. WEB_SPECIFIC         
         LogMessage('      Parse Results Role:'+Role+'||');
         AddMoviePerson(Name,'','',PersonURL,ctDirectors);
         LogMessage('      Get results Director:#'+IntToStr(index)+'|'+Name+'|'+PersonURL+'||ctDirectors');
         curPos:=PosFrom('<td class="name">',ItemList,curPos)                                            //String which opens the subList data. WEB_SPECIFIC
         index:=index+1;
      End;
    End;