Join us
@keskad ・ Sep 12,2022 ・ 1 min read ・ 4921 views
This would be a short story of a case, that happened to me. One of many development teams had reported synchronization failures in ArgoCD, after checking it we discovered that ArgoCD’s repo-server Pod was restarting a few times. Reason? OOMKilled, pretty popular case.
But why an ArgoCD instance with 20 applications had this issue, while ArgoCD instance with 200+ applications didn’t? The difference lies in the usage.
Both ArgoCD instances are using a monorepo, but the affected one is using Kustomize in half of its applications, after checking the in-container memory usage I discovered that multiple instances of Kustomize are taking approximately 80 MB of memory for each process. Given there are 10 applications refreshed simultaneously, then the memory requirements flies up to an astounding level just for a few seconds, but that’s enough to grab a SIGKILL signal from the OOM Killer, that guy does not have mercy.
Everytime, when a commit is pushed to the repository the ArgoCD cache for that repository is wiped and rebuilt again — to build the cache ArgoCD needs to run Helm, Kustomize and other tools to render the manifests.
The solution
There are two steps which could be taken — thanks to ArgoCD’s flexibility and to it’s documentation.
argocd-repo-server fork/exec config management tool to generate manifests. The fork can fail due to lack of memory and limit on the number of OS threads. The — parallelismlimit flag controls how many manifests generations are running concurrently and allows avoiding OOM kills.
In case you are using ArgoCD operator that does not allow appending commandline arguments to repo-server you can set an environment variable instead: ARGOCD_REPO_SERVER_PARALLELISM_LIMIT
2. Additionally to increase the performance of how the ArgoCD discovers changes in a monorepo we can set an annotation on Application level, to make ArgoCD aware of which commited files belongs to which Application so it will refresh state only of applications affected by the files actually commited
annotations:
# “.” resolves to path specified in .spec.source.path
argocd.argoproj.io/manifest-generate-paths: .
Join other developers and claim your FAUN account now!
Platform Engineer
@keskadInfluence
Total Hits
Posts
Only registered users can post comments. Please, login or signup.