How to add full text search to my static website?

Quan Tong

"Life is all about sharing. If we are good at something, pass it on." - Mary Berry

How to add full text search to my static website?

2020-12-30

Categories: Programming

After building the website, I want to add search function. There are some ways to do it:

ElasticSearch: setup and maintenance cost
MongoDB: have to insert posts into database
Google Custom Search Engine: ads?

So, is there any search engine written in Go?

Bleve
riot

Bleve has more star and be maintained more often than riot, I would like to give it a try first.

Backend

Looking at the documentation, there are three steps to add search to your website:

Open a new index:

func createOrOpenIndex(posts []*blog.Post, indexPath string) (bleve.Index, error) {
	var index bleve.Index
	if _, err := os.Stat(indexPath); os.IsNotExist(err) {
		index, err = bleve.NewUsing(indexPath, bleve.NewIndexMapping(), scorch.Name, scorch.Name, nil)
		if err != nil {
			return nil, errors.Errorf("failed to create index at %s: %v", indexPath, err)
		}
	} else if err == nil {
		index, err = bleve.OpenUsing(indexPath, nil)
		if err != nil {
			return nil, errors.Errorf("failed to open index at %s: %v", indexPath, err)
		}

		if err := deletePostsFromIndex(posts, index); err != nil {
			return nil, err
		}
	}
	return index, nil
}

Index will be created when starting, so it must be updated if there is a post has been deleted:

func deletePostsFromIndex(posts []*blog.Post, index bleve.Index) error {
	count, err := index.DocCount()
	if err != nil {
		return errors.Errorf("failed to get number of documents in the index: %v", err)
	}

	query := bleve.NewMatchAllQuery()
	searchRequest := bleve.NewSearchRequest(query)
	searchRequest.Size = int(count)
	searchResults, err := index.Search(searchRequest)
	if err != nil {
		return errors.Errorf("failed to find all documents in the index: %v", err)
	}
	for i := 0; i < len(searchResults.Hits); i++ {
		uri := searchResults.Hits[i].ID
		if !isContains(posts, uri) {
			if err := index.Delete(uri); err != nil {
				return err
			}
		}
	}
	return nil
}

Index posts:

func indexPost(post *blog.Post, batch *bleve.Batch) error {
	doc := document.Document{
		ID: post.URI,
	}
	if err := bleve.NewIndexMapping().MapDocument(&doc, post); err != nil {
		return errors.Errorf("failed to map document: %v", err)
	}

	var b bytes.Buffer
	enc := gob.NewEncoder(&b)
	if err := enc.Encode(post); err != nil {
		return errors.Errorf("failed to encode post: %v", err)
	}

	field := document.NewTextFieldWithIndexingOptions("_source", nil, b.Bytes(), document.StoreField)
	if err := batch.IndexAdvanced(doc.AddField(field)); err != nil {
		return errors.Errorf("failed to add index to the batch: %v", err)
	}
	return nil
}

Search:

func (ps *postService) Search(value string) ([]*blog.Post, error) {
	query := bleve.NewMatchQuery(value)
	request := bleve.NewSearchRequest(query)
	request.Fields = []string{"_source"}
	searchResults, err := ps.index.Search(request)
	if err != nil {
		return nil, errors.Errorf("failed to execute a search request: %v", err)
	}

	var searchPosts []*blog.Post
	for _, result := range searchResults.Hits {
		var post *blog.Post
		b := bytes.NewBuffer([]byte(fmt.Sprintf("%v", result.Fields["_source"])))
		dec := gob.NewDecoder(b)
		if err = dec.Decode(&post); err != nil {
			return nil, errors.Errorf("failed to decode post: %v", err)
		}
		searchPosts = append(searchPosts, post)
	}

	return searchPosts, nil
}

bleve CLI can be used to query the index:

$ bleve query posts.bleve "linux"
5 matches, showing 1 through 5, took 625.866µs
    1. 2019/09/19/about (0.318970)
	Content
		…developer. Then my passionate on Linux and open source led me to a different direction, a system administrator.
With a strong background in Linux system administration, I was gradually moving to backe…
    2. 2019/10/30/docker-exited-127 (0.233462)
	Content
		…/span>
<span style="margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#495050">5</span>COPY build/linux/&lt;binary&gt; .
<span style="margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#495050">6</span>
…

Frontend

Add a search box using forms:

<form class="form-inline my-2 my-lg-0" action="/search">
    <input class="form-control mr-sm-2" type="search" placeholder="Search" aria-label="Search" name="q">
    <button class="btn btn-outline-success my-2 my-sm-0" type="submit">Search</button>
</form>

The action attribute defines where the data gets sent:

	s.router.HandleFunc("/search", s.Error(s.searchHandler))

searchHandler will:

parse form
get the form value
perform search
render posts

References:

Tags: bleve full-text search golang

Edit on GitHub