Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helm chart for transfer.sh #255

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ ADD . /go/src/github.com/dutchcoders/transfer.sh
WORKDIR /go/src/github.com/dutchcoders/transfer.sh

ENV GO111MODULE=on
ENV APP_PORT=8080

# build & install server
RUN go get -u ./... && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags -a -tags netgo -ldflags '-w -extldflags "-static"' -o /go/bin/transfersh github.com/dutchcoders/transfer.sh
Expand All @@ -20,6 +21,6 @@ LABEL maintainer="Andrea Spacca <[email protected]>"
COPY --from=build /go/bin/transfersh /go/bin/transfersh
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

ENTRYPOINT ["/go/bin/transfersh", "--listener", ":8080"]
ENTRYPOINT /go/bin/transfersh --listener :$APP_PORT

EXPOSE 8080
55 changes: 55 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,61 @@ For easy deployment, we've created a Docker container.
```bash
docker run --publish 8080:8080 dutchcoders/transfer.sh:latest --provider local --basedir /tmp/
```
Pass the params to the transfer.sh binary inside container by the *args*, not through docker environment vars.

```bash
docker run -p 8080:8080 dutchcoders/transfer.sh:latest --provider s3 --http-auth-user my-username --http-auth-pass somepassword --aws-access-key $AWS_ACCESS_KEY_ID --aws-secret-key $AWS_SECRET_ACCESS_KEY --bucket $AWS_TRANSFERSH_BUCKET --s3-region $AWS_TRANSFERSH_BUCKET_REGION
```

## Manually run inside kubernetes cluster

```sh
# run locally
kubectl run transfersh --restart=Never --image=dutchcoders/transfer.sh:latest -- --http-auth-user my-username --http-auth-pass somepassword --provider local --basedir=/tmp

# run with s3
kubectl run transfersh --restart=Never --image=dutchcoders/transfer.sh:latest -- --http-auth-user my-username --http-auth-pass somepassword --provider s3 --aws-access-key $AWS_ACCESS_KEY_ID --aws-secret-key $AWS_SECRET_ACCESS_KEY --bucket $AWS_TRANSFERSH_BUCKET --s3-region $AWS_TRANSFERSH_BUCKET_REGION

# Example to manually create needed secrets for deployment params totally aligned with [Usage Params](https://github.com/dutchcoders/transfer.sh#usage-1)
kubectl create secret generic transfersh-secrets --from-literal=HTTP_AUTH_USER=$HTTP_AUTH_USER --from-literal=HTTP_AUTH_PASS=$HTTP_AUTH_PASS --from-literal=AWS_ACCESS_KEY=$AWS_ACCESS_KEY --from-literal=AWS_SECRET_KEY=$AWS_SECRET_KEY --from-literal=BUCKET=$BUCKET --from-literal=S3_REGION=$S3_REGION --from-literal=PROXY_PATH=$PROXY_PATH --from-literal=PROVIDER=$PROVIDER
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed I don't need a secret is needed, but just a config map.
anyway, that's not the problem: any solution we chose should be taken care of by the helm template, not from some previous setup directly in k8s

Copy link
Author

@beyondszine beyondszine Aug 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

umm, this is just for an example of how a secret addition can be done as AWS_ACCESS & AWS_SECRET are things people may not want to put in configmap. But anyway, the option to put both configMap/secret is given by the values.yaml.

Addition of params & their values in configmap can also be given inthe readme.
Idea of putting it in REAMDE was to enable people to also be able to manually do deployment,service,secret/configmap in their cluster Other than automated helm-chart deployment.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem is not being a secret or a config map
the problem is that in order to use the helm chart I have to create the secret directly in k8s
this should be taken care of by the chart itself

```

### TIPS
If your service is going to run behind nginx or any other proxy in your kubernetes cluster then passing on `proxy-path` variable becomes a must to avoid to avoid errors by webend, by-default it is blank.
dont add prefix '/' for the path.
Ex: if your kubernetes ingress piece of routing yaml is like this
```yaml
...
spec:
rules:
- host: api.myhost.mysite.com
http:
paths:
- backend:
serviceName: transfersh
servicePort: 80
path: /filemanager
...
```
=> PROXY_PATH arg must be set to 'filemanager' & not /filemanager.


## Helm chart
```sh
cd charts/transfersh
helm install --debug --name=transfersh transfersh/
```

### NOTE:
- All variables are same as mentioned [here](https://github.com/dutchcoders/transfer.sh#usage-1) with below mentioned operations done on them.
- Operations applied on Usage params
- UPPERCASING them
- replacing hyphens by underscores
- Ex: http-auth-user => HTTP_AUTH_USER, s3-region => S3_REGION
- Every arg needed by the transfer.sh binary is passed via environment variable in deployment yaml injected via the secrets/configmaps at runtime.
- Deployment fails in case of non-availability of secrets/configMaps in your cluster, as selected via values.yaml file.



## S3 Usage

Expand Down
21 changes: 21 additions & 0 deletions charts/transfersh/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
5 changes: 5 additions & 0 deletions charts/transfersh/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Kubernetes
name: transfersh
version: 0.1.0
19 changes: 19 additions & 0 deletions charts/transfersh/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
1. Get the application URL by running these commands:
{{- if .Values.ingress.enabled }}
{{- range .Values.ingress.hosts }}
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ . }}{{ $.Values.ingress.path }}
{{- end }}
{{- else if contains "NodePort" .Values.service.type }}
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "transfersh.fullname" . }})
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w {{ include "transfersh.fullname" . }}'
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "transfersh.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo http://$SERVICE_IP:{{ .Values.service.port }}
{{- else if contains "ClusterIP" .Values.service.type }}
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "transfersh.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl port-forward $POD_NAME 8080:80
{{- end }}
32 changes: 32 additions & 0 deletions charts/transfersh/templates/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "transfersh.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "transfersh.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "transfersh.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
58 changes: 58 additions & 0 deletions charts/transfersh/templates/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: {{ include "transfersh.fullname" . }}
labels:
app.kubernetes.io/name: {{ include "transfersh.name" . }}
helm.sh/chart: {{ include "transfersh.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app.kubernetes.io/name: {{ include "transfersh.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "transfersh.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
{{- $keyRefSource := .Values.argValues.source.type -}}
{{ $keyRefName := .Values.argValues.source.name }}
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
{{- range .Values.argValues.paramNames }}
- name: {{ . | upper | replace "-" "_" }}
valueFrom:
{{ $keyRefSource }}:
name: {{ $keyRefName }}
key: {{ . | upper | replace "-" "_"}}
{{- end }}
args: [ {{- range .Values.argValues.paramNames }} {{ printf "%s%s" $.Values.argIdentifier . | quote}},{{ printf "%s%s%s" "$(" . ")" | upper | replace "-" "_" | quote}},{{- end }} ]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the moment, assuming we will add the missing supported params, it should work, because currently no param is mutually exclusive to each other
tomorrow this would not be true, anyway: we may have conflicting params

also, even if I've not tested, for bool param having the flag set could be enough to enable them: so passing them by default should be avoided, and rather passed or not according to their value

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I guess I understand what you mean.
But isn't the job of params validation & their compatibility should be taken care by the binary itself and not the wrapping frameworks(docker,CI/CD,kubernetes etc.).
If the params have conflict, shouldn't they be just passed through from outside to the binary and let binary handle them & give appropriate error.

For bool values, you point makes sense. But since our param names are not indicating if they are bool or not, then there must be a way in helm layer to know the difference between a bool param & otherwise.
May be you can give me some hint on some you would want Values.yaml to be(specifically declaring params), so that I can backtrack & understand better.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the binary does params validation: if you pass --provider local without --basedir it won't start
tomorrow it could be the case that if you pass params related to S3 when using local provider will fail validation as well: if the chart just passes everything it won't work or will enable functionality that use doesn't want to enable (like with the bool flags)

I'm not such an expert of helm so I don't know what's the most idiomatic way to solve this issue: I guess having different values.yaml for every different provider can be a starting point. for bool flags we could use flag.enabled like you did for the ingress and then pass --set flag.enabled=true to helm

people that has a stable/immutable deployment needing can write their own custom values.yaml, others can rely on the default provided ones and the --set flag

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get it now, what you mean here & above.
Give me some time, let me get back to you on how I think what you say, can be implemented (only as a plan) & if we agree, then probably I can fix changes in code.

livenessProbe:
httpGet:
path: /
port: {{ .Values.service.internalPort }}
readinessProbe:
httpGet:
path: /
port: {{ .Values.service.internalPort }}
resources:
{{ toYaml .Values.resources | indent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{ toYaml . | indent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{ toYaml . | indent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{ toYaml . | indent 8 }}
{{- end }}
38 changes: 38 additions & 0 deletions charts/transfersh/templates/ingress.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{{- if .Values.ingress.enabled -}}
{{- $fullName := include "transfersh.fullname" . -}}
{{- $ingressPath := .Values.ingress.path -}}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ $fullName }}
labels:
app.kubernetes.io/name: {{ include "transfersh.name" . }}
helm.sh/chart: {{ include "transfersh.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- with .Values.ingress.annotations }}
annotations:
{{ toYaml . | indent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ . | quote }}
http:
paths:
- path: {{ $ingressPath }}
backend:
serviceName: {{ $fullName }}
servicePort: http
{{- end }}
{{- end }}
19 changes: 19 additions & 0 deletions charts/transfersh/templates/service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "transfersh.fullname" . }}
labels:
app.kubernetes.io/name: {{ include "transfersh.name" . }}
helm.sh/chart: {{ include "transfersh.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.externalPort }}
targetPort: {{ .Values.service.internalPort }}
protocol: TCP
name: {{ .Values.service.name }}
selector:
app.kubernetes.io/name: {{ include "transfersh.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
63 changes: 63 additions & 0 deletions charts/transfersh/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
replicaCount: 1
image:
repository: dutchcoders/transfer.sh
tag: latest
pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: ""

service:
type: ClusterIP
name: transfersh-service
externalPort: 80
internalPort: 8080

argIdentifier: "--"

argValues : {
"source" :{
"type" : "secretKeyRef",
"name" : "transfersh-secrets"
},
"paramNames" :[
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a lot of parameters missing

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

umm, I couldn't get what you mean here.
The idea why I wrote this is,
whatever params are put in this array of 'paramNames' will be reflected into the secrets automatically(because of the loop). I only put a few for example.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but then either this is a POC or it only supports S3 provider
for me in order for the branch to be merged should support every provider/flag that transfer.sh supports

"http-auth-user",
"http-auth-pass",
"provider",
"aws-access-key",
"aws-secret-key",
"bucket",
"s3-region",
"proxy-path"
]
}

ingress:
enabled: false
annotations: {}
path: /
hosts:
- transfersh.local
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local

resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}