Partial De-duplication in R based on string value match
I have a dataframe named 'reviews' like this:
score_phrase title score release_year release_month release_day
1 Amazing LittleBigPlanet PS Vita 9 2012 9 12
2 Amazing LittleBigPlanet PS Vita -- Marvel Super Hero Edition 9 2012 9 12
3 Great Splice: Tree of Life 8.5 2012 9 12
4 Great NHL 13 8.5 2012 9 11
5 Great NHL 13 8.5 2012 9 11
6 Good Total War Battles: Shogun 7 2012 9 11
7 Awful Double Dragon: Neon 3 2012 9 11
8 Amazing Guild Wars 2 9 2012 9 11
9 Awful Double Dragon: Neon 3 2012 9 11
10 Good Total War Battles: Shogun 7 2012 9 11
Objective: Slight mismatch/typo in column values cause duplication in records. Here Row 1 and Row 2 are duplicates and Row 2 should be dropped after de-duplication.
I used dedup() function of 'SCRUBR' package to perform de-duplication but on a large dataset, I get incorrect number of duplicates when I toggle tolerance level for string matching.
For example:
partial_dup_data <- reviews[1:100,] %>% dedup(tolerance = 0.7)
#count w/o duplicates: 90
attr(partial_dup_data, "dups")
# count of identified duplicates: 16
Could somebody suggest what I am doing incorrectly? Is there another approach to achieve the objective?
r string duplicates matching fuzzy
asked Nov 15 '18 at 4:04
Priya YadavPriya Yadav
133
add a comment |
I have a dataframe named 'reviews' like this:
score_phrase title score release_year release_month release_day
1 Amazing LittleBigPlanet PS Vita 9 2012 9 12
2 Amazing LittleBigPlanet PS Vita -- Marvel Super Hero Edition 9 2012 9 12
3 Great Splice: Tree of Life 8.5 2012 9 12
4 Great NHL 13 8.5 2012 9 11
5 Great NHL 13 8.5 2012 9 11
6 Good Total War Battles: Shogun 7 2012 9 11
7 Awful Double Dragon: Neon 3 2012 9 11
8 Amazing Guild Wars 2 9 2012 9 11
9 Awful Double Dragon: Neon 3 2012 9 11
10 Good Total War Battles: Shogun 7 2012 9 11
Objective: Slight mismatch/typo in column values cause duplication in records. Here Row 1 and Row 2 are duplicates and Row 2 should be dropped after de-duplication.
I used dedup() function of 'SCRUBR' package to perform de-duplication but on a large dataset, I get incorrect number of duplicates when I toggle tolerance level for string matching.
For example:
partial_dup_data <- reviews[1:100,] %>% dedup(tolerance = 0.7)
#count w/o duplicates: 90
attr(partial_dup_data, "dups")
# count of identified duplicates: 16
Could somebody suggest what I am doing incorrectly? Is there another approach to achieve the objective?
r string duplicates matching fuzzy
asked Nov 15 '18 at 4:04
Priya YadavPriya Yadav
133
I don't think you're doing anything wrong. Fuzzy-matching on strings is just by definition, "fuzzy." You could look into various string-distance packages which deal with this sort of issue. All probabilistic matching has a 'grey area' where results are unclear. Often the solution is manual human intervention for the difficult cases.
– thelatemail
Nov 15 '18 at 4:14
also please provide us minimal reproducible example so that we can help you further
– Hunaidkhan
Nov 15 '18 at 4:15
I used the dataset available at Kaggle. Data set : ign.csv kaggle.com/rtatman/data-cleaning-challenge-deduplication/data
– Priya Yadav
Nov 15 '18 at 15:29
add a comment |
I have a dataframe named 'reviews' like this:
score_phrase title score release_year release_month release_day
1 Amazing LittleBigPlanet PS Vita 9 2012 9 12
2 Amazing LittleBigPlanet PS Vita -- Marvel Super Hero Edition 9 2012 9 12
3 Great Splice: Tree of Life 8.5 2012 9 12
4 Great NHL 13 8.5 2012 9 11
5 Great NHL 13 8.5 2012 9 11
6 Good Total War Battles: Shogun 7 2012 9 11
7 Awful Double Dragon: Neon 3 2012 9 11
8 Amazing Guild Wars 2 9 2012 9 11
9 Awful Double Dragon: Neon 3 2012 9 11
10 Good Total War Battles: Shogun 7 2012 9 11
Objective: Slight mismatch/typo in column values cause duplication in records. Here Row 1 and Row 2 are duplicates and Row 2 should be dropped after de-duplication.
I used dedup() function of 'SCRUBR' package to perform de-duplication but on a large dataset, I get incorrect number of duplicates when I toggle tolerance level for string matching.
For example:
partial_dup_data <- reviews[1:100,] %>% dedup(tolerance = 0.7)
#count w/o duplicates: 90
attr(partial_dup_data, "dups")
# count of identified duplicates: 16
Could somebody suggest what I am doing incorrectly? Is there another approach to achieve the objective?
r string duplicates matching fuzzy
asked Nov 15 '18 at 4:04
Priya YadavPriya Yadav
133
I have a dataframe named 'reviews' like this:
score_phrase title score release_year release_month release_day
1 Amazing LittleBigPlanet PS Vita 9 2012 9 12
2 Amazing LittleBigPlanet PS Vita -- Marvel Super Hero Edition 9 2012 9 12
3 Great Splice: Tree of Life 8.5 2012 9 12
4 Great NHL 13 8.5 2012 9 11
5 Great NHL 13 8.5 2012 9 11
6 Good Total War Battles: Shogun 7 2012 9 11
7 Awful Double Dragon: Neon 3 2012 9 11
8 Amazing Guild Wars 2 9 2012 9 11
9 Awful Double Dragon: Neon 3 2012 9 11
10 Good Total War Battles: Shogun 7 2012 9 11
Objective: Slight mismatch/typo in column values cause duplication in records. Here Row 1 and Row 2 are duplicates and Row 2 should be dropped after de-duplication.
I used dedup() function of 'SCRUBR' package to perform de-duplication but on a large dataset, I get incorrect number of duplicates when I toggle tolerance level for string matching.
For example:
partial_dup_data <- reviews[1:100,] %>% dedup(tolerance = 0.7)
#count w/o duplicates: 90
attr(partial_dup_data, "dups")
# count of identified duplicates: 16
Could somebody suggest what I am doing incorrectly? Is there another approach to achieve the objective?
r string duplicates matching fuzzy
r string duplicates matching fuzzy
asked Nov 15 '18 at 4:04
Priya YadavPriya Yadav
133
asked Nov 15 '18 at 4:04
Priya YadavPriya Yadav
133
edited Nov 15 '18 at 15:26
Priya Yadav
asked Nov 15 '18 at 4:04
Priya YadavPriya Yadav
133
asked Nov 15 '18 at 4:04
Priya YadavPriya Yadav
133
asked Nov 15 '18 at 4:04
Priya YadavPriya Yadav
133
133
I don't think you're doing anything wrong. Fuzzy-matching on strings is just by definition, "fuzzy." You could look into various string-distance packages which deal with this sort of issue. All probabilistic matching has a 'grey area' where results are unclear. Often the solution is manual human intervention for the difficult cases.
– thelatemail
Nov 15 '18 at 4:14
also please provide us minimal reproducible example so that we can help you further
– Hunaidkhan
Nov 15 '18 at 4:15
I used the dataset available at Kaggle. Data set : ign.csv kaggle.com/rtatman/data-cleaning-challenge-deduplication/data
– Priya Yadav
Nov 15 '18 at 15:29
add a comment |
I don't think you're doing anything wrong. Fuzzy-matching on strings is just by definition, "fuzzy." You could look into various string-distance packages which deal with this sort of issue. All probabilistic matching has a 'grey area' where results are unclear. Often the solution is manual human intervention for the difficult cases.
– thelatemail
Nov 15 '18 at 4:14
also please provide us minimal reproducible example so that we can help you further
– Hunaidkhan
Nov 15 '18 at 4:15
I used the dataset available at Kaggle. Data set : ign.csv kaggle.com/rtatman/data-cleaning-challenge-deduplication/data
– Priya Yadav
Nov 15 '18 at 15:29
I don't think you're doing anything wrong. Fuzzy-matching on strings is just by definition, "fuzzy." You could look into various string-distance packages which deal with this sort of issue. All probabilistic matching has a 'grey area' where results are unclear. Often the solution is manual human intervention for the difficult cases.
– thelatemail
Nov 15 '18 at 4:14
I don't think you're doing anything wrong. Fuzzy-matching on strings is just by definition, "fuzzy." You could look into various string-distance packages which deal with this sort of issue. All probabilistic matching has a 'grey area' where results are unclear. Often the solution is manual human intervention for the difficult cases.
– thelatemail
Nov 15 '18 at 4:14
also please provide us minimal reproducible example so that we can help you further
– Hunaidkhan
Nov 15 '18 at 4:15
also please provide us minimal reproducible example so that we can help you further
– Hunaidkhan
Nov 15 '18 at 4:15
I used the dataset available at Kaggle. Data set : ign.csv kaggle.com/rtatman/data-cleaning-challenge-deduplication/data
– Priya Yadav
Nov 15 '18 at 15:29
I used the dataset available at Kaggle. Data set : ign.csv kaggle.com/rtatman/data-cleaning-challenge-deduplication/data
– Priya Yadav
Nov 15 '18 at 15:29
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53312272%2fpartial-de-duplication-in-r-based-on-string-value-match%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53312272%2fpartial-de-duplication-in-r-based-on-string-value-match%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I don't think you're doing anything wrong. Fuzzy-matching on strings is just by definition, "fuzzy." You could look into various string-distance packages which deal with this sort of issue. All probabilistic matching has a 'grey area' where results are unclear. Often the solution is manual human intervention for the difficult cases.
– thelatemail
Nov 15 '18 at 4:14
also please provide us minimal reproducible example so that we can help you further
– Hunaidkhan
Nov 15 '18 at 4:15
I used the dataset available at Kaggle. Data set : ign.csv kaggle.com/rtatman/data-cleaning-challenge-deduplication/data
– Priya Yadav
Nov 15 '18 at 15:29