Odtnhj

I am working on a simple full text inverted index trying to build an index of words that I extract from PDF files. I am using PDFBox library to achieve this.

However, I would like to know how does one define a definition of word to index.The way my indexing works is define every word with a space is a word token. For example,

This string, is a code.

In this case: the index table would contain

This
string,
is
a
code.

The flaw here is for like string, , it comes with a comma where I think string would just be sufficient enough because nobody searches string, or code.

Back to my question, is there a specific rule there I could use to define my word token in a way to prevent this kind of issue with what I have ?

Code:

File folder = new File("D:\PDF1");
File listOfFiles = folder.listFiles();

for (File file : listOfFiles) 
 if (file.isFile()) 
 HashSet<String> uniqueWords = new HashSet<>();
 String path = "D:\PDF1\" + file.getName();
 try (PDDocument document = PDDocument.load(new File(path))) 
 if (!document.isEncrypted()) 
 PDFTextStripper tStripper = new PDFTextStripper();
 String pdfFileInText = tStripper.getText(document);
 String lines = pdfFileInText.split("\r?\n");
 for(String line : lines) 
 String words = line.split(" "); 
 for (String word : words) 
 uniqueWords.add(word); 
 

 
 
 catch (IOException e) 
 System.err.println("Exception while trying to read pdf document - " + e);
 
 

Yes. You can use replaceAll method to get rid of non-word characters like this:

uniqueWords.add(word.replaceAll("([\W]+$)|(^[\W]+)", "")); 

If you wanted to remove all punctuation you could do:

for(String word : words) 
 uniqueWords.add(word.replaceAll("[.,!?]", ""));

Which will replace all periods, commas, exclamation marks, and question marks.

If you also want to get rid of quotes you can do:

uniqueWords.add(word.replaceAll("[.,?!"]", "")