Ignoring Edge NGrams in some queries
I have followed this guide to create an auto-completion search... what I am doing is to use Edge NGram tokenizer, to tokenize one of my fields (title). I used these Edge NGrams for my auto-completion search this is my solution.
The auto-complete works fine... but it has broken my search functionality... because now when I use my main search (not the auto-complete) it still searches the NGrams... for example, if I search for "Ski", the result would contain both Ski and Skirt. For my main search, I don't want Skirt to be a match when searching Ski.
Is it possible to configure Elasticsearch to ignore Edge NGrams in some queries and use them in other queries?
elasticsearch asked Nov 12 at 4:19
Hooman
3,0813730
add a comment |
I have followed this guide to create an auto-completion search... what I am doing is to use Edge NGram tokenizer, to tokenize one of my fields (title). I used these Edge NGrams for my auto-completion search this is my solution.
The auto-complete works fine... but it has broken my search functionality... because now when I use my main search (not the auto-complete) it still searches the NGrams... for example, if I search for "Ski", the result would contain both Ski and Skirt. For my main search, I don't want Skirt to be a match when searching Ski.
Is it possible to configure Elasticsearch to ignore Edge NGrams in some queries and use them in other queries?
elasticsearch asked Nov 12 at 4:19
Hooman
3,0813730
is your main search happens when user press enter on search bar ?
– Amit Khandelwal
Nov 12 at 6:42
Yes, I have a search textbox where I don't want the Edge NGrams. I have another auto-complete textbox on a different page where I want to use the Egde NGrams.
– Hooman
Nov 12 at 6:52
ohh great, then its a same use case, pl refer my answer below and lert me know if you have any doubts.
– Amit Khandelwal
Nov 12 at 6:53
add a comment |
I have followed this guide to create an auto-completion search... what I am doing is to use Edge NGram tokenizer, to tokenize one of my fields (title). I used these Edge NGrams for my auto-completion search this is my solution.
The auto-complete works fine... but it has broken my search functionality... because now when I use my main search (not the auto-complete) it still searches the NGrams... for example, if I search for "Ski", the result would contain both Ski and Skirt. For my main search, I don't want Skirt to be a match when searching Ski.
Is it possible to configure Elasticsearch to ignore Edge NGrams in some queries and use them in other queries?
elasticsearch asked Nov 12 at 4:19
Hooman
3,0813730
I have followed this guide to create an auto-completion search... what I am doing is to use Edge NGram tokenizer, to tokenize one of my fields (title). I used these Edge NGrams for my auto-completion search this is my solution.
The auto-complete works fine... but it has broken my search functionality... because now when I use my main search (not the auto-complete) it still searches the NGrams... for example, if I search for "Ski", the result would contain both Ski and Skirt. For my main search, I don't want Skirt to be a match when searching Ski.
Is it possible to configure Elasticsearch to ignore Edge NGrams in some queries and use them in other queries?
elasticsearch elasticsearch asked Nov 12 at 4:19
Hooman
3,0813730
asked Nov 12 at 4:19
Hooman
3,0813730
edited Nov 12 at 4:44
asked Nov 12 at 4:19
Hooman
3,0813730
asked Nov 12 at 4:19
Hooman
3,0813730
asked Nov 12 at 4:19
Hooman
3,0813730
3,0813730
is your main search happens when user press enter on search bar ?
– Amit Khandelwal
Nov 12 at 6:42
Yes, I have a search textbox where I don't want the Edge NGrams. I have another auto-complete textbox on a different page where I want to use the Egde NGrams.
– Hooman
Nov 12 at 6:52
ohh great, then its a same use case, pl refer my answer below and lert me know if you have any doubts.
– Amit Khandelwal
Nov 12 at 6:53
add a comment |
is your main search happens when user press enter on search bar ?
– Amit Khandelwal
Nov 12 at 6:42
Yes, I have a search textbox where I don't want the Edge NGrams. I have another auto-complete textbox on a different page where I want to use the Egde NGrams.
– Hooman
Nov 12 at 6:52
ohh great, then its a same use case, pl refer my answer below and lert me know if you have any doubts.
– Amit Khandelwal
Nov 12 at 6:53
is your main search happens when user press enter on search bar ?
– Amit Khandelwal
Nov 12 at 6:42
is your main search happens when user press enter on search bar ?
– Amit Khandelwal
Nov 12 at 6:42
Yes, I have a search textbox where I don't want the Edge NGrams. I have another auto-complete textbox on a different page where I want to use the Egde NGrams.
– Hooman
Nov 12 at 6:52
Yes, I have a search textbox where I don't want the Edge NGrams. I have another auto-complete textbox on a different page where I want to use the Egde NGrams.
– Hooman
Nov 12 at 6:52
ohh great, then its a same use case, pl refer my answer below and lert me know if you have any doubts.
– Amit Khandelwal
Nov 12 at 6:53
ohh great, then its a same use case, pl refer my answer below and lert me know if you have any doubts.
– Amit Khandelwal
Nov 12 at 6:53
add a comment |
1 Answer
1
active
oldest
votes
I had a similar requirement but in my application auto-complete is different than the main search. auto-completion happens as user is typing and main search happens when user press enter.
Auto-complete and main search results are different in our application as its a requirement in your case. Also it's a general practice.
How I solved this use case :-
For autocomplete instead of index time, we used the https://www.elastic.co/guide/en/elasticsearch/guide/master/prefix-query.html there are pros and cons b/w doing this index time and query time.
For main search it doesn't use the prefix search, hence partial search doesn't happen.
Example
1. If title field contains foo, bar, hello, world than prefix search will bring the result for f, fo but not for oo. If you want oo to be searchable then you need to use 2 different fields for autocomplete and main search that's the only solution.
Otherwise just change your query type to prefix in case of autocomplete and in this case, your index size will also be very less.
We did some testing on if we use edge-N gram token then it was increasing our index size to be doubled, which was not a choice for us as we have very large indexes.
Edit
If you want to have a substring search, for term runni you want title New Balance Running Shoes should come in search result then I would advise you to create 2 fields like title_autocomplete which uses edge n-gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
answered Nov 12 at 6:46
Amit Khandelwal
4,29021433
Hi Amit, thanks for your answer... in my case, auto-complete field, is the title of an Ad on an e-commerce website. So Imagine the title is: "New Balance Running Shoes". Would a prefix query still work if user types: "Runni" or does it have to start from beginning of the term?
– Hooman
Nov 12 at 7:07
@Hooman , it won't work forRunni. as you expected it has to start from the beginning. In this case, its a substring search and then you have to create edge n gram tokens which you already created, but just create 2 fields like title_autocomplete which uses edge n gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
– Amit Khandelwal
Nov 12 at 8:08
Thanks Amit, sounds like using a second field is my best option.
– Hooman
Nov 12 at 8:46
@Hooman, yeah this is how we also solved some of our use cases. Also if you like my answer please don't forget to accept and upvote it :).
– Amit Khandelwal
Nov 12 at 9:12
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53255916%2fignoring-edge-ngrams-in-some-queries%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I had a similar requirement but in my application auto-complete is different than the main search. auto-completion happens as user is typing and main search happens when user press enter.
Auto-complete and main search results are different in our application as its a requirement in your case. Also it's a general practice.
How I solved this use case :-
For autocomplete instead of index time, we used the https://www.elastic.co/guide/en/elasticsearch/guide/master/prefix-query.html there are pros and cons b/w doing this index time and query time.
For main search it doesn't use the prefix search, hence partial search doesn't happen.
Example
1. If title field contains foo, bar, hello, world than prefix search will bring the result for f, fo but not for oo. If you want oo to be searchable then you need to use 2 different fields for autocomplete and main search that's the only solution.
Otherwise just change your query type to prefix in case of autocomplete and in this case, your index size will also be very less.
We did some testing on if we use edge-N gram token then it was increasing our index size to be doubled, which was not a choice for us as we have very large indexes.
Edit
If you want to have a substring search, for term runni you want title New Balance Running Shoes should come in search result then I would advise you to create 2 fields like title_autocomplete which uses edge n-gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
answered Nov 12 at 6:46
Amit Khandelwal
4,29021433
Hi Amit, thanks for your answer... in my case, auto-complete field, is the title of an Ad on an e-commerce website. So Imagine the title is: "New Balance Running Shoes". Would a prefix query still work if user types: "Runni" or does it have to start from beginning of the term?
– Hooman
Nov 12 at 7:07
@Hooman , it won't work forRunni. as you expected it has to start from the beginning. In this case, its a substring search and then you have to create edge n gram tokens which you already created, but just create 2 fields like title_autocomplete which uses edge n gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
– Amit Khandelwal
Nov 12 at 8:08
Thanks Amit, sounds like using a second field is my best option.
– Hooman
Nov 12 at 8:46
@Hooman, yeah this is how we also solved some of our use cases. Also if you like my answer please don't forget to accept and upvote it :).
– Amit Khandelwal
Nov 12 at 9:12
add a comment |
I had a similar requirement but in my application auto-complete is different than the main search. auto-completion happens as user is typing and main search happens when user press enter.
Auto-complete and main search results are different in our application as its a requirement in your case. Also it's a general practice.
How I solved this use case :-
For autocomplete instead of index time, we used the https://www.elastic.co/guide/en/elasticsearch/guide/master/prefix-query.html there are pros and cons b/w doing this index time and query time.
For main search it doesn't use the prefix search, hence partial search doesn't happen.
Example
1. If title field contains foo, bar, hello, world than prefix search will bring the result for f, fo but not for oo. If you want oo to be searchable then you need to use 2 different fields for autocomplete and main search that's the only solution.
Otherwise just change your query type to prefix in case of autocomplete and in this case, your index size will also be very less.
We did some testing on if we use edge-N gram token then it was increasing our index size to be doubled, which was not a choice for us as we have very large indexes.
Edit
If you want to have a substring search, for term runni you want title New Balance Running Shoes should come in search result then I would advise you to create 2 fields like title_autocomplete which uses edge n-gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
answered Nov 12 at 6:46
Amit Khandelwal
4,29021433
Hi Amit, thanks for your answer... in my case, auto-complete field, is the title of an Ad on an e-commerce website. So Imagine the title is: "New Balance Running Shoes". Would a prefix query still work if user types: "Runni" or does it have to start from beginning of the term?
– Hooman
Nov 12 at 7:07
@Hooman , it won't work forRunni. as you expected it has to start from the beginning. In this case, its a substring search and then you have to create edge n gram tokens which you already created, but just create 2 fields like title_autocomplete which uses edge n gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
– Amit Khandelwal
Nov 12 at 8:08
Thanks Amit, sounds like using a second field is my best option.
– Hooman
Nov 12 at 8:46
@Hooman, yeah this is how we also solved some of our use cases. Also if you like my answer please don't forget to accept and upvote it :).
– Amit Khandelwal
Nov 12 at 9:12
add a comment |
I had a similar requirement but in my application auto-complete is different than the main search. auto-completion happens as user is typing and main search happens when user press enter.
Auto-complete and main search results are different in our application as its a requirement in your case. Also it's a general practice.
How I solved this use case :-
For autocomplete instead of index time, we used the https://www.elastic.co/guide/en/elasticsearch/guide/master/prefix-query.html there are pros and cons b/w doing this index time and query time.
For main search it doesn't use the prefix search, hence partial search doesn't happen.
Example
1. If title field contains foo, bar, hello, world than prefix search will bring the result for f, fo but not for oo. If you want oo to be searchable then you need to use 2 different fields for autocomplete and main search that's the only solution.
Otherwise just change your query type to prefix in case of autocomplete and in this case, your index size will also be very less.
We did some testing on if we use edge-N gram token then it was increasing our index size to be doubled, which was not a choice for us as we have very large indexes.
Edit
If you want to have a substring search, for term runni you want title New Balance Running Shoes should come in search result then I would advise you to create 2 fields like title_autocomplete which uses edge n-gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
answered Nov 12 at 6:46
Amit Khandelwal
4,29021433
I had a similar requirement but in my application auto-complete is different than the main search. auto-completion happens as user is typing and main search happens when user press enter.
Auto-complete and main search results are different in our application as its a requirement in your case. Also it's a general practice.
How I solved this use case :-
For autocomplete instead of index time, we used the https://www.elastic.co/guide/en/elasticsearch/guide/master/prefix-query.html there are pros and cons b/w doing this index time and query time.
For main search it doesn't use the prefix search, hence partial search doesn't happen.
Example
1. If title field contains foo, bar, hello, world than prefix search will bring the result for f, fo but not for oo. If you want oo to be searchable then you need to use 2 different fields for autocomplete and main search that's the only solution.
Otherwise just change your query type to prefix in case of autocomplete and in this case, your index size will also be very less.
We did some testing on if we use edge-N gram token then it was increasing our index size to be doubled, which was not a choice for us as we have very large indexes.
Edit
If you want to have a substring search, for term runni you want title New Balance Running Shoes should come in search result then I would advise you to create 2 fields like title_autocomplete which uses edge n-gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
answered Nov 12 at 6:46
Amit Khandelwal
4,29021433
edited Nov 13 at 15:14
answered Nov 12 at 6:46
Amit Khandelwal
4,29021433
answered Nov 12 at 6:46
Amit Khandelwal
4,29021433
answered Nov 12 at 6:46
Amit Khandelwal
4,29021433
4,29021433
Hi Amit, thanks for your answer... in my case, auto-complete field, is the title of an Ad on an e-commerce website. So Imagine the title is: "New Balance Running Shoes". Would a prefix query still work if user types: "Runni" or does it have to start from beginning of the term?
– Hooman
Nov 12 at 7:07
@Hooman , it won't work forRunni. as you expected it has to start from the beginning. In this case, its a substring search and then you have to create edge n gram tokens which you already created, but just create 2 fields like title_autocomplete which uses edge n gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
– Amit Khandelwal
Nov 12 at 8:08
Thanks Amit, sounds like using a second field is my best option.
– Hooman
Nov 12 at 8:46
@Hooman, yeah this is how we also solved some of our use cases. Also if you like my answer please don't forget to accept and upvote it :).
– Amit Khandelwal
Nov 12 at 9:12
add a comment |
Hi Amit, thanks for your answer... in my case, auto-complete field, is the title of an Ad on an e-commerce website. So Imagine the title is: "New Balance Running Shoes". Would a prefix query still work if user types: "Runni" or does it have to start from beginning of the term?
– Hooman
Nov 12 at 7:07
@Hooman , it won't work forRunni. as you expected it has to start from the beginning. In this case, its a substring search and then you have to create edge n gram tokens which you already created, but just create 2 fields like title_autocomplete which uses edge n gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.
– Amit Khandelwal
Nov 12 at 8:08
Thanks Amit, sounds like using a second field is my best option.
– Hooman
Nov 12 at 8:46
@Hooman, yeah this is how we also solved some of our use cases. Also if you like my answer please don't forget to accept and upvote it :).
– Amit Khandelwal
Nov 12 at 9:12
Hi Amit, thanks for your answer... in my case, auto-complete field, is the title of an Ad on an e-commerce website. So Imagine the title is: "New Balance Running Shoes". Would a prefix query still work if user types: "Runni" or does it have to start from beginning of the term?
– Hooman
Nov 12 at 7:07
Hi Amit, thanks for your answer... in my case, auto-complete field, is the title of an Ad on an e-commerce website. So Imagine the title is: "New Balance Running Shoes". Would a prefix query still work if user types: "Runni" or does it have to start from beginning of the term?
– Hooman
Nov 12 at 7:07
@Hooman , it won't work for
Runni . as you expected it has to start from the beginning. In this case, its a substring search and then you have to create edge n gram tokens which you already created, but just create 2 fields like title_autocomplete which uses edge n gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.– Amit Khandelwal
Nov 12 at 8:08
@Hooman , it won't work for
Runni . as you expected it has to start from the beginning. In this case, its a substring search and then you have to create edge n gram tokens which you already created, but just create 2 fields like title_autocomplete which uses edge n gram analyzer and used for autocomplete query and title which uses some other analyzer like standard or something else which solves your main search use case.– Amit Khandelwal
Nov 12 at 8:08
Thanks Amit, sounds like using a second field is my best option.
– Hooman
Nov 12 at 8:46
Thanks Amit, sounds like using a second field is my best option.
– Hooman
Nov 12 at 8:46
@Hooman, yeah this is how we also solved some of our use cases. Also if you like my answer please don't forget to accept and upvote it :).
– Amit Khandelwal
Nov 12 at 9:12
@Hooman, yeah this is how we also solved some of our use cases. Also if you like my answer please don't forget to accept and upvote it :).
– Amit Khandelwal
Nov 12 at 9:12
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53255916%2fignoring-edge-ngrams-in-some-queries%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
is your main search happens when user press enter on search bar ?
– Amit Khandelwal
Nov 12 at 6:42
Yes, I have a search textbox where I don't want the Edge NGrams. I have another auto-complete textbox on a different page where I want to use the Egde NGrams.
– Hooman
Nov 12 at 6:52
ohh great, then its a same use case, pl refer my answer below and lert me know if you have any doubts.
– Amit Khandelwal
Nov 12 at 6:53