curl - solution for https

[VVV_Easy_Programing]

I think that the problem it that you program path have spaces and the 'del' command don't be able to erase 'downpage.htm' (it tries, wait 1 seg and tries, tries ...)
Perhaps curl may the same problem.

This afternoon I shall try something with inverted commas but ¿may you do a try with a copy of PVdB in a folder without spaces, for instance: C:\Users\Ivo\Documents\PersonalVideoDB\?

[Ivek23]

This afternoon I shall try something with inverted commas but ¿may you do a try with a copy of PVdB in a folder without spaces, for instance: C:\Users\Ivo\Documents\PersonalVideoDB\?

Yes, I made a change in the way as mentioned above. Unfortunately, the result is still the same as before.

[VVV_Easy_Programing]

I made and attach the next version:
VERSION:    0.3.1.0 (06/11/2016) -> VVV: Use quotation marks for the paths.

but I see some things bizarre in you log file:

(8.11.2016 8:46:09) PVD Version: 0.9.9.21

¿why you don't use the latest version 1.0.2.7?

(8.11.2016 8:46:37) UpdateToolbar: 4
(8.11.2016 8:46:45) GET: http://www.videodb.info/upload/check.php
(8.11.2016 8:46:47) Update error: Socket Error # 10061
Connection refused.
(8.11.2016 8:46:52) UpdateToolbar: 5

It seems that your copy of PVdB have problems even with the http pages ¿are you sure that you haven't a proxy, firewall or antivirus with your PVdB? It's very rare that curl works out of PVdB and not in ¿are you sure that Proxomitron is not active?.

¿Can you test it in a clean downloaded 1.0.2.7 version? You can use a portable one decompressing:
http://www.videodb.info/bin/pvd1.zip
in C:\Users\Ivo\Documents\PersonalVideoDB\ and adding the three needed curl files.
Perhaps other people can help us trying the script.

BTW:
The biggest solved of script is synchronize PVdB and the curl download and be sure when you parse that 'downpage.htm' is fresh and completed. So I do two steeps:
1) First the script delete the saved 'downpage.htm'.

    //Delete the ancient dowloaded page file. Needed for wait to curl download.       
    While FileExists(ScriptPath+BASE_DOWNLOAD_FILE) do begin
         FileExecute('cmd.exe', '/C del '+ScriptPath+BASE_DOWNLOAD_FILE);
         LogMessage('      Waiting 1s for delete:'+ScriptPath+BASE_DOWNLOAD_FILE);
         wait (1000);
    end;

2) After curl call it waits to detected the  'downpage.htm'

    //Wait curl finish and exist the downloaded page
    While Not(FileExists(ScriptPath+BASE_DOWNLOAD_FILE)) do begin
         LogMessage('      Waiting 2s for exists of:'+ScriptPath+BASE_DOWNLOAD_FILE);
         wait (2000);
    end;

So in your first log, the script has problems to delete 'downpage.htm', it tries and wait, tries and wait:

(8.11.2016 8:47:03)       Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:04)       Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:05)       Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm

When you block this part of code it detects that something goes wrong:

(8.11.2016 8:54:58) Global Var-ScriptPath|C:\Program Files\Personal Video Database\Scripts\|
(8.11.2016 8:54:58)       Some error deleting: C:\Program Files\Personal Video Database\Scripts\downpage.htm

 

[Ivek23]

(8.11.2016 8:46:09) PVD Version: 0.9.9.21

¿why you don't use the latest version 1.0.2.7?

I am also used, but as the basis I have loaded on the PC version 0.9.9.21. A big reason why is basically not used primarily this that once with this version use a particular database, the latter version rearranges and is no longer usable with old 0.9.9.21 version or the old. 0.9.9.21 version can no longer open. For certain things when using a larger number of databases (at least for me so) old 0.9.9.21 version is better, easier to edit certain things rather than the latest version.

(8.11.2016 8:46:37) UpdateToolbar: 4
(8.11.2016 8:46:45) GET: http://www.videodb.info/upload/check.php
(8.11.2016 8:46:47) Update error: Socket Error # 10061
Connection refused.
(8.11.2016 8:46:52) UpdateToolbar: 5

It seems that your copy of PVdB have problems even with the http pages ¿are you sure that you haven't a proxy, firewall or antivirus with your PVdB?

My bad, I forgot to change the proxy settings, and then it was the same.

It's very rare that curl works out of PVdB and not in ¿are you sure that Proxomitron is not active?.

Definitely he was not in that the time in use.

¿Can you test it in a clean downloaded 1.0.2.7 version? You can use a portable one decompressing:
http://www.videodb.info/bin/pvd1.zip
in C:\Users\Ivo\Documents\PersonalVideoDB\ and adding the three needed curl files.

I did it and it works perfectly. Even in the old 0.9.9.21. version now works perfectly.


A big thank you for your efforts, which was made in the script, it is now again usable. It will be as already mentioned:

Finally, it's not difficult to adapt this script to another HTTPS page. All people are free to use it.

This script will be a good basis for writing the scripts for the site, which also use a secure https connection and this is not an option the use of Proxomitron.

Once again many thanks for the help that has been a problem with RottenTomatoes script successfully solved. Your help and expertise is welcome in the future in solving similar problems.

Thank you very much.

Perhaps other people can help us trying the script.

I hope so.

[VVV_Easy_Programing]

 

If you arrive, in this script or in other, to need download pictures from a HTTPS link (so binary download), tell me because I think that it will be necessary make some code.

[Ivek23]

If you arrive, in this script or in other, to need download pictures from a HTTPS link (so binary download), tell me because I think that it will be necessary make some code.

Excellent suggestion, I certainly for this proposal and for part of the code, this piece of code in the future would certainly come in handy.

[Ivek23]

When Rottentomatoes_ [HTTPS] script reports that it is necessary to Advanced Search, you can help me to find a specific movie title, amongst other options also Rottentomatoes_simple_Bing_Search script to find the right movie and URL. But there are also all ends because the script can not be transferred RottenTomatoes https URL.

Until now, the URL open in browser and manually copy the PVdB.

You may be in the script to make changes to it using the curl program, the transfer url in PVdB.

The script is attached.

[VVV_Easy_Programing]

It's a good idea to include a Bing search in the script for advanced search but I had tried and this is a very rare case.
A lot of films are found with the script in normal search ¿can you write me the name of the film that needs advanced search?

I think that must be another problem ¿the title of your films are in english?
In the search the script use 'title' in place of traditional 'origtitle', you can change it in the code:

end else begin  //The movie URL not exist, search mode needed. Downloand the search page. 
          DownloadURL:=StringReplace(SEARCH_STR_TRUE,'%s',GetFieldValueXML('title'),True,True,False);DownloadURL:=StringReplace(SEARCH_STR_TRUE,'%s',GetFieldValueXML('title'),True,True,False);

In other side, I put a double security in the script for need advanced search:

        if (0=Pos('Search Results - Rotten Tomatoes', HTML)) or (0<Pos('Sorry, no results found', HTML)) then begin      //WEB_SPECIFIC
          ShowMessage('Not results. Try again or make an advanced search out of the script.'+Chr(13)+'Write the movie URL (http) in the PVdB manually (remember separated by spaces) and run again.','Rotten Tomatoes Search Results');
 

If you have problems with easy films (for instance: godfather) you can delete '(0=Pos('Search Results - Rotten Tomatoes', HTML)) or' in order to try to reduce the false alerts.

I promise you integrate the bing search in the script but I don't have to much time now. I propose you that you finish Rottentomatoes_ [HTTPS] script and I continue the script adding the Bing search.

[Ivek23]

It's a good idea to include a Bing search in the script for advanced search but I had tried and this is a very rare case.
A lot of films are found with the script in normal search ¿can you write me the name of the film that needs advanced search?

Not yet, because I'm still in the initial phase of testing script.

I think that must be another problem ¿the title of your films are in english?
In the search the script use 'title' in place of traditional 'origtitle', you can change it in the code:

end else begin  //The movie URL not exist, search mode needed. Downloand the search page. 
          DownloadURL:=StringReplace(SEARCH_STR_TRUE,'%s',GetFieldValueXML('title'),True,True,False);DownloadURL:=StringReplace(SEARCH_STR_TRUE,'%s',GetFieldValueXML('title'),True,True,False);

This option changes work when "origtitle" in the second line, but does not work if "origtitle" field is empty or if the "origtitle" in the first line, before the change is operated. This means that this change, should not be correct.

In other side, I put a double security in the script for need advanced search:

        if (0=Pos('Search Results - Rotten Tomatoes', HTML)) or (0<Pos('Sorry, no results found', HTML)) then begin      //WEB_SPECIFIC
          ShowMessage('Not results. Try again or make an advanced search out of the script.'+Chr(13)+'Write the movie URL (http) in the PVdB manually (remember separated by spaces) and run again.','Rotten Tomatoes Search Results');
 

If you have problems with easy films (for instance: godfather) you can delete '(0=Pos('Search Results - Rotten Tomatoes', HTML)) or' in order to try to reduce the false alerts.

Is not necessary.

I promise you integrate the bing search in the script but I don't have to much time now. I propose you that you finish Rottentomatoes_ [HTTPS] script and I continue the script adding the Bing search.

Also, when I tested it, I add in this topic. I hope as soon as possible, because currently I do not have much time due to personal matters.

[Ivek23]

I promise you integrate the bing search in the script but I don't have to much time now. I propose you that you finish Rottentomatoes_ [HTTPS] script and I continue the script adding the Bing search.

There is version 0.3.2.0, which has all the basic details of what is, at least to me it seems important to transfer information from a poster and movie info and of course the Cast section.

As with the search but the results are concerned, it is added to the poster and the underlying reason, because once labeled the title of the film you want to open in the browser, then a browser window such as this link.
http://www.videodb.info/forum_en/%20https://www.rottentomatoes.com/m/beverly_hills_cop

Therefore, I added a url link, which the browser opens a Web page labeled the film or the URL if one wants to make sure that the correct address searches search movie.
https://www.rottentomatoes.com/m/beverly_hills_cop

In any event, it should not be used for the transfer from informabij movie site, because it will appear the following message.

IOHandler value is not valid

[Ivek23]

It's a good idea to include a Bing search in the script for advanced search but I had tried and this is a very rare case.
A lot of films are found with the script in normal search ¿can you write me the name of the film that needs advanced search?

Not yet, because I'm still in the initial phase of testing script.

This movie title

Blood Diamond
http://www.rottentomatoes.com/m/blood_diamond

Borderline
http://www.rottentomatoes.com/m/1002913-borderline/

Comanche Territory
http://www.rottentomatoes.com/m/comanche_territory

is found only with the help of advanced search (Rottentomatoes_simple_Bing_Search script), but still can not find the bottom limited changes.

I think that must be another problem ¿the title of your films are in english?
In the search the script use 'title' in place of traditional 'origtitle', you can change it in the code:

end else begin  //The movie URL not exist, search mode needed. Downloand the search page. 
          DownloadURL:=StringReplace(SEARCH_STR_TRUE,'%s',GetFieldValueXML('title'),True,True,False);DownloadURL:=StringReplace(SEARCH_STR_TRUE,'%s',GetFieldValueXML('title'),True,True,False);

This option changes work when "origtitle" in the second line, but does not work if "origtitle" field is empty or if the "origtitle" in the first line, before the change is operated. This means that this change, should not be correct.

[VVV_Easy_Programing]

VERSION:    0.3.3.0 (23/11/2016) -> VVV: Added Advanced Search with Bing (http search engine) and some code improvements.

Preview poster in search list is disable with // because give me problems

Quote for Ivek23: But with those three movies the RT search work for me. I had must try the bing search with a roundabout (not present in the attach script) so test it in real conditions.

[afrocuban]

Also, I got this in non-https rottentomates script:

Code: [Select]

//Get ~mfURL~ or ~RT Url~
endPos := Pos('" rel="canonical" itemprop="url"/>', HTML);
if endPos > 0 then begin
curPos := PrevPos('</title> <link href="', HTML, endPos);
AddFieldValue(mfURL, Copy(HTML, curPos + 21, endPos - curPos - 21));
end else
AddFieldValue(mfURL, MovieURL);
AddCustomFieldValueByName('Rottentomatoes.com', '<link url="' + MovieURL + '">Rottentomatoes.com</link>');

is it possbile to include it in [HTTPS] script?

Thank you once again, you're really doing great job for all of us who can't imagine movies without PVD.

[Ivek23]

Added in final version.

[afrocuban]

Thank you immensely, Ivek!

[Ivek23]

Thank you immensely, Ivek!

Welcome.

[VVV_Easy_Programing]

I have managed to download the HTTPS page from FilmAffinity directly with the Windows PowerShell (no external files needed). I do not know if it will work with all pages and on all windows. Check out my script "FilmAffinity [ES][HTTPS]" to see the code.

[VVV_Easy_Programing]

Happily, we have finished the "Beta" phase in the PVD MOD and move on to "Release" state.
So, we closed this topic now and we continue in the new topic:

Personal Video Database 1.0.2.7 MOD
http://www.videodb.info/forum_en/index.php/topic,4134.0.html

See you all there.