[SOLVED] Multi threaded IMDB fetching?

[Happy2k]

Is there any way to make the IMDB fetcing multi threaded?
Worst part is, if it encounters a movie its not sure about - everything stops. So you cant leave it over night, and just fix the unknown ones when you get up.

If there is a way to speed it up, id very much like to get a hint about it

Else, a possible solution could be:
- Multi threadded IMDB fetching
- A "fix me"-list, for movies that are not recognized 100%

Besides that.. the most recent update seems to have helped on the recognition - its only few that are not automaticly recognized now!

[rick.ca]

If there is a way to speed it up, id very much like to get a hint about it

Silent mode. 

A "fix me"-list, for movies that are not recognized 100%

Use a search to display the records not updated by the Silent mode update. Since all of those will require user input, you'll be kept reasonably busy while updating those with Silent mode off.

Multi threadded IMDB fetching

That's an interesting idea for other reasons (e.g., update speed), but I would be very surprised if IMDb did not ban IP's making multiple simultaneous requests. I'm actually surprised were able to get away with what we are doing.

[deazo]

 Hi Rick,
 
 could you clarify "silent mode" for me please?
 I always thought it was taking the first result in the list of films found, hence that there was no way to return to the "not 100% match".

 But from what you wrote it sounds like it is just skipping the ones it is not sure about?
 Could you confirm please?

 Thanks

[rick.ca]

But from what you wrote it sounds like it is just skipping the ones it is not sure about?

Right. It would make quite a mess of things if guessed.

[Happy2k]

If there is a way to speed it up, id very much like to get a hint about it

Silent mode.  

Yea, but silent isnt faster, is it?

A "fix me"-list, for movies that are not recognized 100%

Use a search to display the records not updated by the Silent mode update. Since all of those will require user input, you'll be kept reasonably busy while updating those with Silent mode off.

That sure is a good possibility - gonna try it.

Multi threadded IMDB fetching

That's an interesting idea for other reasons (e.g., update speed), but I would be very surprised if IMDb did not ban IP's making multiple simultaneous requests. I'm actually surprised were able to get away with what we are doing.

Before using PVD, i was trying out different programs. Most of them had a very fast IMDb fetch. I dont think IMDb would care - ive never heard of IMDb ip banning someone.
I wouldnt mind being a ginniepig for testing out multi threaded IMDb fetching.

[rick.ca]

Yea, but silent isnt faster, is it?

In the context of your original question, it's days faster.

I dont think IMDb would care - ive never heard of IMDb ip banning someone.

Perhaps not, but HTTP error 503 occurs regularly. They probably just mean the server is busy, but it could also mean the server is telling you it's busy because it's getting too many requests from you.

[deazo]

 Thanks for clarification Rick, I thought Silent Mode was taking the first movie found (!), that's why I was never using it.
 In terms of speed, I think the script is very fast or at least fast enough.
 Unless you're on 56K of course 

[Happy2k]

Yea, but silent isnt faster, is it?

In the context of your original question, it's days faster.

I dont think IMDb would care - ive never heard of IMDb ip banning someone.

Perhaps not, but HTTP error 503 occurs regularly. They probably just mean the server is busy, but it could also mean the server is telling you it's busy because it's getting too many requests from you.

I'd say its worth trying - if it isnt alot of work to get it working.
Again, i wouldnt mind testing it.

 Thanks for clarification Rick, I thought Silent Mode was taking the first movie found (!), that's why I was never using it.
 In terms of speed, I think the script is very fast or at least fast enough.
 Unless you're on 56K of course 

Fast enough? Yesterday i added 700 movies. After 1½ hour, it had completed 270 movies.
Others seems to think so too: http://personal-video-database-portable.en.softonic.com/ (Cons: Database internet retrieval can be slow)
One more: http://www.softsea.com/review/IMDB-Movie-Explorer.html (quote: It is dog slow, but does the job)

Its working - which is the primary goal - but i'd just like to see it a bit faster. If that isnt possible, that's ok too

[buah]

I imported 5500 entries from IMDB in a day

[Happy2k]

I imported 5500 entries from IMDB in a day

What is your secret?! I started PVD about 2½ hours ago, and it has now passed 500 entries - in silent mode.

[nostra]

Any multithreaded system is a lot of work and it is very difficult to make it work good with the current plugin system, so I do not see a quick solution here. I will still keep your suggestion in mind and will take a look at the download performance some time in future.

[Happy2k]

Any multithreaded system is a lot of work and it is very difficult to make it work good with the current plugin system, so I do not see a quick solution here. I will still keep your suggestion in mind and will take a look at the download performance some time in future.

Awesome I look forward to that.


Regarding the silent mode.. How do i see which movies it has skipped?

[nostra]

Search for movies without URL, for example.

[Happy2k]

Search for movies without URL, for example.

How do you do that? I can find the URL search in Advanced Search, but i cant see how im gonna search for no URL.

[buah]

I imported 5500 entries from IMDB in a day

What is your secret?! I started PVD about 2½ hours ago, and it has now passed 500 entries - in silent mode.

2,5hrs x 10=25 hrs

500 x 10= 5000

[nostra]

I can find the URL search in Advanced Search, but i cant see how im gonna search for no URL.

Click on the operator (=) and select IS NULL
Or select the containing operator and set keyword to imdb

[rick.ca]

I'd say its worth trying - if it isnt alot of work to get it working. Again, i wouldnt mind testing it.

It is a lot of work. It makes much more sense for you to do the testing, and then make the case for multi-threading. I'm not sure it will work, but you could install multiple copies of the program, and run them simultaneously using the -portable and -open=database switches so each uses it's own configuration and database.

Others seems to think so too: http://personal-video-database-portable.en.softonic.com/ (Cons: Database internet retrieval can be slow) One more: http://www.softsea.com/review/IMDB-Movie-Explorer.html (quote: It is dog slow, but does the job)

You're going to trust an outdated review by an amateur reviewer who tried the program once?! Slow compared to what? The second review is for another program.

[Happy2k]

I can find the URL search in Advanced Search, but i cant see how im gonna search for no URL.

Click on the operator (=) and select IS NULL
Or select the containing operator and set keyword to imdb

Ahhhh there's actually a menu behind "containing" - awesome!

Dang.. 123 out of 712 have been skipped.


rick.ca -> Gonna try that when im re-doing the entire list! Worth trying at least.

If i had trusted only 1 review, i wouldn't have started using PVD I always do alot more research than a single review.
The times where i trust one single review, is when its authored by me (I do hardware reviews)

But i really like PVD - it was hard getting to work properly in the beginning, but the possiblities are endless.

[Happy2k]

I imported 5500 entries from IMDB in a day

What is your secret?! I started PVD about 2½ hours ago, and it has now passed 500 entries - in silent mode.

2,5hrs x 10=25 hrs

500 x 10= 5000

My math tells me two things: 5000 != 5500, and 25 hours != 24 hours
And then there's the skipped ones. But it helps alot, when most can be done automaticly.

[nostra]

But i really like PVD - it was hard getting to work properly in the beginning, but the possiblities are endless.

Unfortunately it seems like many users have difficulties when starting using PVD. If you have suggestions how to make the process easier for beginners, then feel free to post in the Feature Suggestions board.