How to add full text search to my static website?

Quan Tong

"Life is all about sharing. If we are good at something, pass it on." - Mary Berry

How to add full text search to my static website?

2020-12-30

Categories: Programming

After building the website, I want to add search function. There are some ways to do it:

ElasticSearch: setup and maintenance cost
MongoDB: have to insert posts into database
Google Custom Search Engine: ads?

So, is there any search engine written in Go?

Bleve
riot

Bleve has more star and be maintained more often than riot, I would like to give it a try first.

Backend

Looking at the documentation, there are three steps to add search to your website:

Open a new index:

 1func createOrOpenIndex(posts []*blog.Post, indexPath string) (bleve.Index, error) {
 2	var index bleve.Index
 3	if _, err := os.Stat(indexPath); os.IsNotExist(err) {
 4		index, err = bleve.NewUsing(indexPath, bleve.NewIndexMapping(), scorch.Name, scorch.Name, nil)
 5		if err != nil {
 6			return nil, errors.Errorf("failed to create index at %s: %v", indexPath, err)
 7		}
 8	} else if err == nil {
 9		index, err = bleve.OpenUsing(indexPath, nil)
10		if err != nil {
11			return nil, errors.Errorf("failed to open index at %s: %v", indexPath, err)
12		}
13
14		if err := deletePostsFromIndex(posts, index); err != nil {
15			return nil, err
16		}
17	}
18	return index, nil
19}

Index will be created when starting, so it must be updated if there is a post has been deleted:

 1func deletePostsFromIndex(posts []*blog.Post, index bleve.Index) error {
 2	count, err := index.DocCount()
 3	if err != nil {
 4		return errors.Errorf("failed to get number of documents in the index: %v", err)
 5	}
 6
 7	query := bleve.NewMatchAllQuery()
 8	searchRequest := bleve.NewSearchRequest(query)
 9	searchRequest.Size = int(count)
10	searchResults, err := index.Search(searchRequest)
11	if err != nil {
12		return errors.Errorf("failed to find all documents in the index: %v", err)
13	}
14	for i := 0; i < len(searchResults.Hits); i++ {
15		uri := searchResults.Hits[i].ID
16		if !isContains(posts, uri) {
17			if err := index.Delete(uri); err != nil {
18				return err
19			}
20		}
21	}
22	return nil
23}

Index posts:

 1func indexPost(post *blog.Post, batch *bleve.Batch) error {
 2	doc := document.Document{
 3		ID: post.URI,
 4	}
 5	if err := bleve.NewIndexMapping().MapDocument(&doc, post); err != nil {
 6		return errors.Errorf("failed to map document: %v", err)
 7	}
 8
 9	var b bytes.Buffer
10	enc := gob.NewEncoder(&b)
11	if err := enc.Encode(post); err != nil {
12		return errors.Errorf("failed to encode post: %v", err)
13	}
14
15	field := document.NewTextFieldWithIndexingOptions("_source", nil, b.Bytes(), document.StoreField)
16	if err := batch.IndexAdvanced(doc.AddField(field)); err != nil {
17		return errors.Errorf("failed to add index to the batch: %v", err)
18	}
19	return nil
20}

Search:

 1func (ps *postService) Search(value string) ([]*blog.Post, error) {
 2	query := bleve.NewMatchQuery(value)
 3	request := bleve.NewSearchRequest(query)
 4	request.Fields = []string{"_source"}
 5	searchResults, err := ps.index.Search(request)
 6	if err != nil {
 7		return nil, errors.Errorf("failed to execute a search request: %v", err)
 8	}
 9
10	var searchPosts []*blog.Post
11	for _, result := range searchResults.Hits {
12		var post *blog.Post
13		b := bytes.NewBuffer([]byte(fmt.Sprintf("%v", result.Fields["_source"])))
14		dec := gob.NewDecoder(b)
15		if err = dec.Decode(&post); err != nil {
16			return nil, errors.Errorf("failed to decode post: %v", err)
17		}
18		searchPosts = append(searchPosts, post)
19	}
20
21	return searchPosts, nil
22}

bleve CLI can be used to query the index:

 1$ bleve query posts.bleve "linux"
 25 matches, showing 1 through 5, took 625.866µs
 3    1. 2019/09/19/about (0.318970)
 4	Content
 5		…developer. Then my passionate on Linux and open source led me to a different direction, a system administrator.
 6With a strong background in Linux system administration, I was gradually moving to backe…
 7    2. 2019/10/30/docker-exited-127 (0.233462)
 8	Content
 9		…/span>
10<span style="margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#495050">5</span>COPY build/linux/&lt;binary&gt; .
11<span style="margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#495050">6</span>
12…

Frontend

Add a search box using forms:

1<form class="form-inline my-2 my-lg-0" action="/search">
2    <input class="form-control mr-sm-2" type="search" placeholder="Search" aria-label="Search" name="q">
3    <button class="btn btn-outline-success my-2 my-sm-0" type="submit">Search</button>
4</form>

The action attribute defines where the data gets sent:

1	s.router.HandleFunc("/search", s.Error(s.searchHandler))

searchHandler will:

parse form
get the form value
perform search
render posts

References:

Tags: bleve full-text search golang

Edit on GitHub