"Life is all about sharing. If we are good at something, pass it on." - Mary Berry

gocloud - writing data to a bucket: 403

2022-12-23

Problem

We are writing some integration test using Go CDK. After writing some data to a bucket:

1	writer, err := buckOut.NewWriter(ctx, fileDst, nil)
2	if err != nil {
3		logger.Errorf("failed to write to fileDst: %v", err)
4		return err
5	}
6	defer writer.Close()

we got an error when reading:

1(code=NotFound): storage: object doesn't exist

By reading the documentation, I pay attention to this:

Closing the writer commits the write to the provider, flushing any buffers, and releases any resources used while writing, so you must always check the error of Close.

Read More...


Terraform failed to acquire state lock: 403: Access denied., forbidden

2022-11-17

Problem

We stored Terraform state in gcs. For some reasons, we got this error randomly when running on BitBucket pipelines:

 1 2│ Error: Error acquiring the state lock
 3 4│ Error message: 2 errors occurred:
 5* writing
 6"gs://bucket/path/to/default.tflock"
 7│ failed: googleapi: Error 403: Access denied., forbidden
 8* storage: object doesn't exist
 9101112│ Terraform acquires a state lock to protect the state from being written
13│ by multiple users at the same time. Please resolve the issue above and try
14│ again. For most commands, you can disable locking with the "-lock=false"
15│ flag, but this is not recommended.
16

Read More...


Docker Compose healthcheck

2021-09-09

The most important thing when running integration test using docker-compose is ensured that one container is started completely before others.

Sometime wait-for-it is not enough:

 1  cassandra:
 2    image: bitnami/cassandra:latest
 3    ports:
 4      - '7000:7000'
 5      - '9042:9042'
 6    volumes:
 7      - /path/to/init-scripts:/docker-entrypoint-initdb.d
 8
 9  wait-for-cassandra:
10    image: willwill/wait-for-it
11    command: cassandra:9042 -t 60
12    depends_on:
13      - cassandra:

Read More...


condition form of depends_on in docker-compose version 3

2021-03-05

As version 3 no longer supports the condition form of depends_on, what is the alternative way to wait for a container to be started completely?

From 1.27.0, 2.x and 3.x are merged with COMPOSE_SPEC schema.

version is now optional. So, you can just remove it and specify a condition as before:

 1services:
 2  web:
 3    build: .
 4    depends_on:
 5      redis:
 6        condition: service_healthy
 7  redis:
 8    image: redis
 9    healthcheck:
10      test: ["CMD", "redis-cli", "ping"]
11      interval: 1s
12      timeout: 3s
13      retries: 30

Read More...


Let's Encrypt too many certificates already issued

2021-02-04

Traefik is configured to use Let’s Encrypt to generate certificate for my blog (and other services) automatically. One day after restarting, I cannot access to my blog via HTTPS anymore (NET::ERR_CERT_AUTHORITY_INVALID). Why?

By looking at the Traefik logs, I found this:

time=“2021-02-04T01:54:33Z” level=error msg=“Unable to obtain ACME certificate for domains \“quantonganh.com\”: unable to generate a certificate for the domains [quantonganh.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: quantonganh.com: see https://letsencrypt.org/docs/rate-limits/, url: “ providerName=le.acme routerName=blog-secured@docker rule=“Host(quantonganh.com)”

Read More...


plugins/docker failed to resolve Keycloak hostname?

2021-01-27

After integrating Docker registry with Keycloak, the publishing step failed to authenticate with Docker Registry.

The full error message is:

1time="2021-01-26T13:44:18.485121053Z" level=error msg="Handler for POST /v1.40/auth returned error: Get https://docker.domain.com/v2/: Get https://sso.domain.com/auth/realms/application/protocol/docker-v2/auth?account=******&client_id=docker&offline_token=true&service=aws-docker-registry: dial tcp: lookup sso.domain.com on 127.0.0.11:53: no such host"

sso.domain.com is a local hostname which can be resolved on the host. How can I make it resolvable inside the plugins/docker container?

I found some similar issues:

but they are slightly differences.

Look at this: http://plugins.drone.io/drone-plugins/drone-docker/

Read More...


How to run a pipeline step only when pushing to a new branch?

2021-01-07

GitHub sends push hook when pushing to a new branch and merging a PR. How can we distinguish between them?

We are running integration test by triggering a downstream build when a PR is merged into specific branches. In the downstream repository, we pull all the docker images which has been built and pushed from upstream. The thing is we also used the drone-convert-pathschanged to only publish what modules has been changed.

So, what happens when pushing to a new release-* branch?

Read More...


Unable to obtain ACME certificate for domains

2020-12-28

Lets Encrypt tells me that my domain contains an invalid character. What is it?

remark42 is configured like this:

 1  remark42:
 2    image: umputun/remark42:arm64
 3    container_name: "remark42"
 4    restart: always
 5    labels:
 6      - traefik.enable=true
 7      - traefik.http.routers.remark42.rule=Host(`${REMARK_URL}`)
 8      - traefik.http.routers.remark42.entrypoints=https
 9      - traefik.http.routers.remark42.tls.certresolver=le
10      - traefik.http.services.remark42.loadbalancer.server.port=8080

Read More...


A && B || C is not the same as if-then-else

2020-12-26

I have been thought that if-then-else statement can be written in one line by using A && B || C but I was wrong.

To speed up my Drone CI time, I configured the static-check step to only run when there are some .go files have been changed:

1git --no-pager diff --name-only ${DRONE_COMMIT_LINK##*/} | grep -q "\.go$" && golangci-lint run --deadline 2m -v ./... || true

I was thought that A && B || C is the same as:

Read More...


How to perform integration testing in Go?

2020-09-29

Integration testing can be triggered by using Drone downstream plugin:

 1steps:
 2- name: trigger
 3  image: plugins/downstream:linux-amd64
 4  settings:
 5    params:
 6    - COMMIT_BRANCH=${DRONE_COMMIT_BRANCH}
 7    repositories:
 8    - repo/integration-test@${DRONE_COMMIT_BRANCH}
 9    server: https://drone.example.com
10    token:
11      from_secret: drone_token

It can be separated with unit tests by using build tags:

1// +build integration
2
3package webserver_test

Then we can write code to perform integration test as usual.

Read More...