A place to share what I learn/experience

"Life is all about sharing. If we are good at something, pass it on." - Mary Berry

Do not use jq when working with large number

2021-09-07

We use etcd to store application configuration. On production, config is loaded from json file by using Python.

I’m wondering if we can use jq to do that. So, I tried something like this:

1host=${ETCD_HOST:-127.0.0.1}
2port=${ETCD_PORT:-2379}
3for key in $(jq -r 'keys[]' "$1"); do
4  value=$(jq -r ".$key" -c "$1")
5  ETCDCTL_API=3 etcdctl --endpoints="$host:$port" put "$key" "$value"
6done

Config is loaded into etcd but some values are not the same as in the JSON file. What is going on?

fish shell - very high CPU usage

2021-07-30

Sometimes my Macbook is loud like a plow, and the culprit is usually docker. This time it’s different, top -u tells me that fish shell is taking up > 100% CPU.

I just switched from zsh to fish about a year ago after having problems with git completion.

My favorite feature about fish is probably autosuggestions. Normally, when I want to run a command again, I can control+R, type a few letters to select the command I want to run. With fish shell, I can type that command and then -> or control+F is fine.

Troubleshooting slow network file transfer

2021-06-25

We have a Hadoop data cluster. Recently, my colleague noticed that data transfer between servers was slow. Not all are slow; only half of them have problems.

The first tool that comes to mind is iperf3. After installing, I run iperf3 -s then iperf3 -c. While other servers have bandwidth > 900 Mbits/sec, the slow ones are just:

 1[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
 2[  4]   0.00-1.00   sec  11.9 MBytes  99.4 Mbits/sec    0    170 KBytes
 3[  4]   1.00-2.00   sec  11.1 MBytes  93.3 Mbits/sec    0    204 KBytes
 4[  4]   2.00-3.00   sec  10.9 MBytes  91.7 Mbits/sec    0    204 KBytes
 5[  4]   3.00-4.00   sec  8.26 MBytes  69.3 Mbits/sec  102    154 KBytes
 6[  4]   4.00-5.00   sec  10.9 MBytes  91.8 Mbits/sec    3    157 KBytes
 7[  4]   5.00-6.00   sec  10.9 MBytes  91.7 Mbits/sec    0    165 KBytes
 8[  4]   6.00-7.00   sec  10.9 MBytes  91.7 Mbits/sec    0    168 KBytes
 9[  4]   7.00-8.00   sec  10.9 MBytes  91.7 Mbits/sec    0    173 KBytes
10[  4]   8.00-9.00   sec  10.9 MBytes  91.7 Mbits/sec    0    173 KBytes
11[  4]   9.00-10.00  sec  10.9 MBytes  91.7 Mbits/sec    0    173 KBytes
12- - - - - - - - - - - - - - - - - - - - - - - - -
13[ ID] Interval           Transfer     Bandwidth       Retr
14[  4]   0.00-10.00  sec   108 MBytes  90.4 Mbits/sec  105             sender
15[  4]   0.00-10.00  sec   107 MBytes  89.5 Mbits/sec                  receiver

Automate office tasks with Python

2021-06-10

Hàng tháng bạn nhận được một file excel về danh sách D. Từ danh sách này bạn cần filter cột C theo tiêu chí T1 để lấy ra danh sách D1. Có được D1 rồi bạn Save As thành t1.xlsx. Làm xong T1, bạn tiếp tục với T2 -> t2.xlsx … Làm tiếp cho đến Tn -> tn.xlsx. Có được các danh sách t1.xlsx -> tn.xlsx này rồi, giờ bạn cần gửi mail:

t1.xlsx: gửi cho P1, P2, P3

The power of praise

2021-03-18

K, 9 tuổi.

Dạo này đang luyện: cắm cơm, nhặt rau. Hôm trước học bài mới: gọt su su. Bài này hơi khó hơn chút: đầu tiên dùng dao cắt đúng khía thành các miếng nhỏ, sau đó gọt sạch, cuối cùng là bỏ lõi. Cứ mỗi lần làm xong một miếng, K lại hỏi:

Thế này được chưa b?
Chuẩn đấy, b thấy miếng này đẹp hơn miếng trước.

Và đúng là nó đẹp hơn thật. Lời khen đã giúp K làm tốt hơn.

condition form of depends_on in docker-compose version 3

2021-03-05

As version 3 no longer supports the condition form of depends_on, what is the alternative way to wait for a container to be started completely?

From 1.27.0, 2.x and 3.x are merged with COMPOSE_SPEC schema.

version is now optional. So, you can just remove it and specify a condition as before:

 1services:
 2  web:
 3    build: .
 4    depends_on:
 5      redis:
 6        condition: service_healthy
 7  redis:
 8    image: redis
 9    healthcheck:
10      test: ["CMD", "redis-cli", "ping"]
11      interval: 1s
12      timeout: 3s
13      retries: 30

Let's Encrypt too many certificates already issued

2021-02-04

Traefik is configured to use Let’s Encrypt to generate certificate for my blog (and other services) automatically. One day after restarting, I cannot access to my blog via HTTPS anymore (NET::ERR_CERT_AUTHORITY_INVALID). Why?

By looking at the Traefik logs, I found this:

time=“2021-02-04T01:54:33Z” level=error msg=“Unable to obtain ACME certificate for domains \“quantonganh.com\”: unable to generate a certificate for the domains [quantonganh.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: quantonganh.com: see https://letsencrypt.org/docs/rate-limits/, url: “ providerName=le.acme routerName=blog-secured@docker rule=“Host(quantonganh.com)”

plugins/docker failed to resolve Keycloak hostname?

2021-01-27

After integrating Docker registry with Keycloak, the publishing step failed to authenticate with Docker Registry.

The full error message is:

1time="2021-01-26T13:44:18.485121053Z" level=error msg="Handler for POST /v1.40/auth returned error: Get https://docker.domain.com/v2/: Get https://sso.domain.com/auth/realms/application/protocol/docker-v2/auth?account=******&client_id=docker&offline_token=true&service=aws-docker-registry: dial tcp: lookup sso.domain.com on 127.0.0.11:53: no such host"

sso.domain.com is a local hostname which can be resolved on the host. How can I make it resolvable inside the plugins/docker container?

I found some similar issues:

but they are slightly differences.

Look at this: http://plugins.drone.io/drone-plugins/drone-docker/

How to run a pipeline step only when pushing to a new branch?

2021-01-07

GitHub sends push hook when pushing to a new branch and merging a PR. How can we distinguish between them?

We are running integration test by triggering a downstream build when a PR is merged into specific branches. In the downstream repository, we pull all the docker images which has been built and pushed from upstream. The thing is we also used the drone-convert-pathschanged to only publish what modules has been changed.

So, what happens when pushing to a new release-* branch?

How to add full text search to my static website?

2020-12-30

After building the website, I want to add search function. There are some ways to do it:

ElasticSearch: setup and maintenance cost
MongoDB: have to insert posts into database
Google Custom Search Engine: ads?

So, is there any search engine written in Go?

Bleve
riot

Bleve has more star and be maintained more often than riot, I would like to give it a try first.

Backend

Looking at the documentation, there are three steps to add search to your website:

First
«
1
2
3
4
5
6
7
»
Last