Why do we only get ~25% of movies with Wikidata matches? #14

audiodude · 2024-09-27T02:31:04Z

The number might be wrong, but the last I heard we were only getting around 25% of the movies in the Netflix data set to match. This is a bit disappointing. We should look into this and see if anything becomes obvious for why this is the case.

A couple of current theories:

Many of the "movies" are actually TV shows or specials (aka "The Office Season 4").
The years in the Netflix data are DVD publication years, and the "publication date" in Wikidata is the year the movie was released. So if a movie was released in October 2002, but on DVD in March 2003, there could be a discrepancy.

skyfenton · 2024-11-08T02:00:14Z

The publication date is definitely a large part of the issue here. After logging just the first 10 movies, only 3 match (ex. Fighter 2001 should actually match Fighter (2000)). Could partially solve by looking in a range before the given year in case the dvd was released in the year after the movie release, but there could be a case where there's a DVD release of a movie after a couple years (for example if it was only on VHS for a while) which would escape the lookback.

Maybe it's worth looking into other user rating datasets or even scraping our own from somewhere? I know someone posted a link in the discord to a imdb rating set on huggingfaces, though I haven't looked at it yet.

skyfenton · 2024-11-08T04:03:31Z

Discussed solution, likely best to find explicit match rather than assume range when a movie should exist. Good approach would be to prioritize matching titles, then have some kind of database/hash that explicitly matches duplicates.

skyfenton · 2024-12-04T21:10:32Z

Looks like we might actually have a different problem. After manually checking a couple missed matches, it should be matching the correct title and year, but isn't (ex. movie id 7, "8 Man" 1992). I tried removing the year filter and it still wouldn't match, so it must not be matching the title string.

audiodude · 2024-12-05T17:58:38Z

Great work, we should investigate further!

One thing to note is that we're using the "search" functionality for the movie titles. Maybe we should be looking for an exact match?

audiodude · 2024-12-05T18:03:37Z

Here's the query for "8 Man" in the wikidata SPARQL query playground:

Sorry for the huge URL!

We should be able to play with parameters in there to see if we can get a match.

In this case, I think it has something to do with the UNION to find the release date. It looks for movies that have either:

A release date in the United States, or
A release date with no country specified

However it doesn't match movies that have no release date specified at all, which seems to be the case with this movie: https://www.wikidata.org/wiki/Q116548163

audiodude · 2024-12-05T18:09:05Z

How about this?

It adds a clause to the UNION to include movies with no release date at all, and moves the release date filtering into the spots where a release date is found:

            {
                # Get US release date
                ?item p:P577 ?releaseDateStatement .
                ?releaseDateStatement ps:P577 ?releaseDate .
                ?releaseDateStatement pq:P291 wd:Q30 .          
                FILTER (YEAR(?releaseDate) = 1992) .
            }
            UNION
            {
                # Get unspecified release date
                ?item p:P577 ?releaseDateStatement .
                ?releaseDateStatement ps:P577 ?releaseDate .
                FILTER NOT EXISTS { ?releaseDateStatement pq:P291 ?country }         
                FILTER (YEAR(?releaseDate) = 1992) .
            }
            UNION
            {
                # Release date doesn't exist, don't try to match on it
                FILTER NOT EXISTS { ?item p:P577 ?releaseDateStatement }
            }

Of course now we get more "duplicates"/matching results which we have to deal with.

audiodude · 2024-12-05T18:28:03Z

Here's the finished query I came up with, using title/label matches instead of the search service:

https://query.wikidata.org/#SELECT%20%2a%20WHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP31%2Fwdt%3AP279%2a%20wd%3AQ11424%20.%0A%0A%20%20%3Fitem%20rdfs%3Alabel%20%3FitemLabel%20.%0A%20%20FILTER%20%28lang%28%3FitemLabel%29%20%3D%20%22en%22%29%20.%0A%20%20%7B%0A%20%20%20%20%23%20Movies%20with%20the%20title%20in%20the%20title%20property%0A%20%20%20%20%3Fitem%20wdt%3AP1476%20%3Ftitle%20.%0A%20%20%20%20FILTER%28STR%28%3Ftitle%29%20%3D%20%228%20Man%22%29%20.%0A%20%20%7D%0A%20%20UNION%0A%20%20%7B%0A%20%20%20%20%23%20Movies%20where%20the%20title%20matches%20the%20label%0A%20%20%20%20%3Fitem%20rdfs%3Alabel%20%228%20Man%22%40en%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%0A%20%20%7D%0A%0A%20%20%23%20Now%20try%20to%20match%20on%20release%20date%2C%20if%20the%20item%20has%20one%0A%20%20%7B%0A%20%20%20%20%23%20Get%20US%20release%20date%0A%20%20%20%20%3Fitem%20p%3AP577%20%3FreleaseDateStatement%20.%0A%20%20%20%20%3FreleaseDateStatement%20ps%3AP577%20%3FreleaseDate%20.%0A%20%20%20%20%3FreleaseDateStatement%20pq%3AP291%20wd%3AQ30%20.%20%20%20%20%20%20%20%20%20%20%0A%20%20%20%20FILTER%20%28YEAR%28%3FreleaseDate%29%20%3D%201992%29%20.%0A%20%20%7D%0A%20%20UNION%0A%20%20%7B%0A%20%20%20%20%23%20Get%20unspecified%20release%20date%0A%20%20%20%20%3Fitem%20p%3AP577%20%3FreleaseDateStatement%20.%0A%20%20%20%20%3FreleaseDateStatement%20ps%3AP577%20%3FreleaseDate%20.%0A%20%20%20%20FILTER%20NOT%20EXISTS%20%7B%20%3FreleaseDateStatement%20pq%3AP291%20%3Fcountry%20%7D%20%20%20%20%20%20%20%20%20%0A%20%20%20%20FILTER%20%28YEAR%28%3FreleaseDate%29%20%3D%201992%29%20.%0A%20%20%7D%0A%20%20UNION%0A%20%20%7B%0A%20%20%20%20%23%20Release%20date%20doesn%27t%20exist%2C%20don%27t%20try%20to%20match%20on%20it%0A%20%20%20%20FILTER%20NOT%20EXISTS%20%7B%20%3Fitem%20p%3AP577%20%3FreleaseDateStatement%20%7D%0A%20%20%7D%0A%0A%20%20OPTIONAL%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP136%20%3Fgenre%20.%0A%20%20%20%20%3Fgenre%20rdfs%3Alabel%20%3FgenreLabel%20.%0A%20%20%20%20FILTER%20%28lang%28%3FgenreLabel%29%20%3D%20%22en%22%29%20.%0A%20%20%7D%0A%0A%20%20OPTIONAL%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP57%20%3Fdirector.%0A%20%20%20%20%3Fdirector%20rdfs%3Alabel%20%3FdirectorLabel.%0A%20%20%20%20FILTER%20%28lang%28%3FdirectorLabel%29%20%3D%20%22en%22%29%0A%20%20%7D%0A%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20.%20%7D%0A%7D%0A

PTAL!

audiodude · 2024-12-05T20:35:55Z

It seems that last query has problems with Lilo and Stitch -> Lilo & Stitch for example.

skyfenton · 2024-12-11T04:56:15Z

Hmm.... To be fair, other sources (like letterboxd) will likely be correct or link an id to another source like The Movie Database, which would be crossreferenced on wikidata so it's not totally crucial to match titles.

audiodude · 2024-12-12T21:59:48Z

Whether we match to Wikidata or "other sources like letterboxd", we will still need to perform something akin to Named Entity Recognition to identify the movie in those systems based on just a title and a year.

audiodude assigned gaferty Sep 27, 2024

skyfenton self-assigned this Dec 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why do we only get ~25% of movies with Wikidata matches? #14

Why do we only get ~25% of movies with Wikidata matches? #14

audiodude commented Sep 27, 2024

skyfenton commented Nov 8, 2024 •

edited

Loading

skyfenton commented Nov 8, 2024

skyfenton commented Dec 4, 2024 •

edited

Loading

audiodude commented Dec 5, 2024

audiodude commented Dec 5, 2024

audiodude commented Dec 5, 2024

audiodude commented Dec 5, 2024

audiodude commented Dec 5, 2024

skyfenton commented Dec 11, 2024

audiodude commented Dec 12, 2024

Why do we only get ~25% of movies with Wikidata matches? #14

Why do we only get ~25% of movies with Wikidata matches? #14

Comments

audiodude commented Sep 27, 2024

skyfenton commented Nov 8, 2024 • edited Loading

skyfenton commented Nov 8, 2024

skyfenton commented Dec 4, 2024 • edited Loading

audiodude commented Dec 5, 2024

audiodude commented Dec 5, 2024

audiodude commented Dec 5, 2024

audiodude commented Dec 5, 2024

audiodude commented Dec 5, 2024

skyfenton commented Dec 11, 2024

audiodude commented Dec 12, 2024

skyfenton commented Nov 8, 2024 •

edited

Loading

skyfenton commented Dec 4, 2024 •

edited

Loading