Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A Docker Container, tool for prezto development #1753

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
*.zwc.old
modules/*/cache.zsh
contrib
.container
24 changes: 24 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
FROM alpine as prezto-devel
LABEL maintainer="[email protected]"
RUN apk --no-cache update && apk upgrade && \
Copy link
Contributor

@casaper casaper Feb 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of --no-cache the common recommendation is this at the end

I've just noticed that --no-cache you renders the clean-up part I wanted to mention obsolete.

And Why do you do apk upgrade ? Since you imply alpine:latest in FROM.
Because If you didn't do that, it would probably shrink the image size significantly, without you having any flaws I can see.

apk add util-linux pciutils usbutils coreutils binutils\
findutils grep man man-pages mdocml-apropos less less-doc \
make grep zsh zsh-vcs zsh-zftp zsh-calendar zsh-doc git \
vim git-zsh-completion tmux tmux-zsh-completion tree \
docker-zsh-completion
RUN addgroup prezto && adduser -D prezto -G prezto -S /bin/zsh
WORKDIR /home/prezto
USER prezto
RUN cd /home/prezto && mkdir src
COPY . src
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you used ADD instead of COPY it would speed up things, because then the files would not really be copied into the container.
Or do you need it like that so that your local sources won't be changed with the later make runs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please tell me how ADD will speed things up? According to docker's best practices individual COPY statements are preferred.

Copy link
Contributor

@casaper casaper Feb 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADD adds the directory as a docker volume.
COPY really creates copies of the files into the docker image.

At least that was how I understood it so far...

Copy link
Contributor

@wadkar wadkar Feb 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it works like that. Neither statements add a directory as a volume. Docker volumes are different directives manipulated by VOLUME statement in Dockerfile or -v flag at runtime.

Both ADD and COPY statements copy the source from the build context (first argument) to the destination in docker image (second argument).

Specifically, as the best practices link explains, the ADD statement has additional magic to it - it will automagically uncompress a compressed file, or if the first argument is a URL it will download it.

COPY . src is good first pass for a Dockerfile. Though ideally, one would prefer to have separate statements for each directory, so that a change in one file in one directory will not invalidate the entire cache. But then we run into the problem of how we will order individual COPY statements. So, I am okay with COPY . src for now.

Edit: Just saw your below comments. At first glance, I like the volumes approach with the caveat that modification from within docker container will modify the host file. I'll check out this PR over the weekend and give it a shot.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well however, it doesn't really matter to me actually if ADD or COPY is used in the end here.

My main point is not running stuff in the image building process, but rather run it on the finished image...
Its just quicker, and sllicker. If you prefer to build the image newly each time. you run something, or change code and then test it, and fail and change it again... up to you. 😉

RUN cp src/Makefile .
RUN make .clone
RUN make src .homercs
Copy link
Contributor

@casaper casaper Feb 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I don't understand is why you run make inside the docker file.

Without fully understanding your makefile and your shell script, I just try to tell you how I would approach that what I assume you want to achieve. If that assumptions are wrong or if I'm anyhow wrong, no big deal. I only try to help.

My Dockerfile would be like this:

FROM alpine

RUN apk --no-cache add \
  util-linux \
  pciutils \
  usbutils \
  coreutils \
  binutils \
  findutils \
  grep \
  man \
  man-pages \
  mdocml-apropos \
  less \
  less-doc \
  make \
  grep \
  zsh \
  zsh-vcs \
  zsh-zftp \
  zsh-calendar \
  zsh-doc \
  git \
  vim \
  git-zsh-completion \
  tmux \
  tmux-zsh-completion \
  tree \
  docker-zsh-completion \
  && addgroup prezto \
  && adduser -D prezto -G prezto -S /bin/zsh

WORKDIR /home/prezto
USER prezto

CMD ["/bin/zsh"]

And then I would add a docker-compose.yml file and run the stuff through that:

version: "3.7"

services:
  prezto:
    build: .
    volumes:
      - ./:/home/prezto/.zprezto
      - ./my-test-stuff:/home/prezto/test
      - ./runcoms/zshenv:/home/prezto/.zshenv
      - ./runcoms/zprofile:/home/prezto/.zprofile
      - ./runcoms/zshrc:/home/prezto/.zshrc
      - ./runcoms/zpreztorc:/home/prezto/.zpreztorc
      - ./runcoms/zlogin:/home/prezto/.zlogin
      - ./runcoms/zlogout:/home/prezto/.zlogout

Now you can set that up with:

docker-compose build  # builds your docker file
docker-compose run --rm prezto /bin/zsh
Couldn't read file /home/prezto/.zprezto/modules/prompt/functions/prompt_agnoster_setup containing theme agnoster.
Couldn't read file /home/prezto/.zprezto/modules/prompt/functions/prompt_powerlevel10k_setup containing theme powerlevel10k.
Couldn't read file /home/prezto/.zprezto/modules/prompt/functions/prompt_powerlevel9k_setup containing theme powerlevel9k.
Couldn't read file /home/prezto/.zprezto/modules/prompt/functions/prompt_powerline_setup containing theme powerline.
Couldn't read file /home/prezto/.zprezto/modules/prompt/functions/prompt_pure_setup containing theme pure.
prompt_sorin_setup:7: async: function definition file not found
prompt_sorin_async_tasks:4: command not found: async_start_worker
prompt_sorin_async_tasks:5: command not found: async_register_callback
prompt_sorin_async_tasks:10: command not found: async_flush_jobs
prompt_sorin_async_tasks:13: command not found: async_job
~ ❯❯❯

I did exactly this here, and it is partial C&P from my shell. So I'm sure that it works.

This has several benefits:

  1. your image will not have to check out the prezto repository, because that you allready have in your dev dir
  2. /home/prezto has all the dotfiles like .zshenv allready in there, directly from your dev machines runcoms
  3. You can use the same image, it's smaller, and you only need to create new containers

You can have some test scripts inside the prezto project like this:

# remember that i mounted `./my-test-stuff:/home/prezto/test` in the docker-compose.yml
docker-compose run --rm prezto /home/prezto/test/some-script.sh
I'm echoed by  some-script.sh doing the stuff you want to do...

This way you can then run technically anything in that container and get that container flushed right after your done.
And you're working with your current WIP directories code.

I just recon it is not really the way to run your tests within the Dockerfile's RUN statements.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeap, this makes a lot more sense than building prezto inside the image.

RUN rm Makefile
# ENTRYPOINT ["/bin/zsh", "-"]
# CMD ["/bin/zsh","-l","-o","verbose","-o","xtrace","-o","sourcetrace"]
CMD ["/bin/zsh","-l"]




113 changes: 113 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
#!/usr/bin/make shebang
# _ ____ _
# _ __ _ __ ___ ___| |_ ___ | _ \ ___ ___| | _____ _ __
# | '_ \| '__/ _ \_ / __/ _ \ _____| | | |/ _ \ / __| |/ / _ \ '__|
# | |_) | | | __// /| || (_) |_____| |_| | (_) | (__| < __/ |
# | .__/|_| \___/___|\__\___/ |____/ \___/ \___|_|\_\___|_|
# |_|
# [email protected] 20191025 004745 -0700 PDT 1571989665 d(-_- )b...
# This makefile automates image building and container management
# for easy development of prezto and prezto-plugins without messing
# up your current configuration. Prezto is installed fresh on an
# alpine container, where you may add your edits
#
#######################################################################
# REPO and IMG define the TAG for our image. this is how it gets named
# when pushed to dockerhum. REPO should be your dockerhub username
# IMG would be nice if its related to the project.
REPO = hlecuanda
IMG = prezto-dev
TAG = $(REPO)/$(IMG)
# ALLCODE recirds all code files so we can have the docker image
# depend on all of them. if any file changes, the image gets rebuilt
ALLCODE != find . -type f | grep -vEe '^\./\.+|*.md'
RUNCOMS != ls runcoms
# The context is the current directory. this is important if you have
# a buildserver, v.gr on gitlab or google cloud. its still needed for
# local builds. so don't touch this, unless you know what you're doing
CONTEXT = .
# All reciipes on this makefile are zsh scripts, not bash or sh
# scripts, so we don't have to be context-switching
SHELL = zsh
# This removes the default build rules for gmake's built in suffixes
# when debugging the Makefile, output is a lot more readable, plus
# if you're not making C programs, it just makes sense to avoid
# mistery bugs
.SUFFIXES =

# We don't want to fail recipies on error for the targets if .IGNORE
.IGNORE: clean realclean ;

# These targets don't create files and are defined for convenience.
# note that other targets do create real files which are used to
# determine the state of the build
.PHONY: default run clean realclean ;

# an empty recipe, to avoid expensive operations if we "make" by
# mistake. it can be made to depend on another target and it will
# become the target of a "make" call with no arguments, vg.r:
# default: image; results in "make" and "make image" doing the same
default: ;

# a user callable target, depends on .container/image whiech records
# the timestamp of the last successfuly built image
image: .container/image ;

# we declare .container/image to depend on $(ALLCODE) which contains
# all files in the distribution. thus, if ANY file changes, it makes
# our image obsolete and will be re-made on next make command
.container/image: $(ALLCODE)
docker build -t $(TAG) $(CONTEXT)
[[ ! -d .container ]] && mkdir .container || :
touch $@

# NOTE: The following targets ( .homercs clone and .clone ) are
# run inside the container while building a container image. They are
# called from the Dockerfile when you "make image". .homercs creates
# the dotfiles on our home directory and depends on .clone, so .clone
# should be made before .homercs. We're spelling these out explicitly
# for clarity instead of using Makefile enchantments that would make
# this a 2 line recipe
.homercs: .clone
ln -s .zprezto/runcoms/zshenv /home/prezto/.zshenv
ln -s .zprezto/runcoms/zprofile /home/prezto/.zprofile
ln -s .zprezto/runcoms/zshrc /home/prezto/.zshrc
ln -s .zprezto/runcoms/zpreztorc /home/prezto/.zpreztorc
ln -s .zprezto/runcoms/zlogin /home/prezto/.zlogin
ln -s .zprezto/runcoms/zlogout /home/prezto/.zlogout
touch $@

# clones the prezto repository we have copied from the build context
# into the container, into a .zprezto directory, as the installation
# instructions recommend.
.clone:
git clone --recursive src .zprezto
touch $@

# This runs an interactive (-it) ephemeral (--rm) container named
# $(IMG) created from the image stored at $(TAG) it is made to depend
# on container/image so that it checks whether image is up to # date.
# Given that image depends on $(ALLCODE), calling "make run" will use
# the latest image, unless any code file hasa changed in which case,
# the image is rebuilt and then run.
run: .container/image
docker run -it --rm -h $(IMG) --name $(IMG) $(TAG)

# Removes the current container and image, so we can build a new one
# from scratch. If you want a real clean slate, then "make realclean"
clean:
docker container rm $(TAG)
docker image rm $(TAG)
rm -fv .container/*

# Deep cleaning, will remove dangling (intermediate build) images
# and containers for which a final image or container cannot be found
# (i.e has been deleted by clean or removed after usage for ephemeral
# containers. Run every once in a while.
realclean:
$(MAKE) clean
docker rmi $(TAG)
docker container prune
docker image prune

# vim: set ft=make sw=2 tw=7 fdm=manual noet :
94 changes: 94 additions & 0 deletions container-README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Prezto Docker Container

This branch contains a Dockerfile and a Makefile that hopefully may
prove helpful for prezto development.

The idea is to have a totally independent and isolated environemnet
in which to test changes to either prezto-core or any plugin that you
may be working on, **without disrupting your environment**, thus enabling
automated testing and even continuous integration.

This is a proof of concept, it may not be even a good idea to have
this on the main prezto repository. on the other hand, the container
image should depend on the code, so if the team finds this to be a
useful tool, there are choices to be made in that respect. My
intention is to introduce this as a helpful tool for development and
for new users to try prezto easely

Here is a screencast showing what the container can do so far

[![asciicast](https://asciinema.org/a/277054.svg)](https://asciinema.org/a/277054)

The container is a basic install of [alpine linux](https://alpinelinux.org) so the download
is reasonably small at around 200M, since debian based images can
weigh in around 1.5G.

On the container we have a few utilities and additional software that
prezto has core support for, (tmux, make, etc) and you can try it i
easily by running:

```bash
docker pull hlecuanda/prezto-dev:latest
```

once you have the image, create a container from it:

```bash
docker run -it --rm -h prezto hlecuanda/prezto-dev:latest
```

That will create an interactive (`--it`) ephemeral container (`--rm`)
whose hostname is prezto (`-h prezto`) based on the aforementioned
imag. you should be sitting at the plain sorin prompt in a brand new
prezto instance.

A development and testing workflow can be achieved by mounting a
project's directory on to the image's filesystem:

```bash
cd /path/to/project/root
docker run -it --rm -h prezto \
-v $(pwd):/home/prezto/.zprezto/modules/${:-$(pwd):t} \
hlecuanda/prezto-dev:latest
```

This will mount the current directory on the container's filesystem,
you can develop on your own machine and environnment, and test your
changes running on the container, your actual source will already be
in-place on prezto's directory hierarchy, as if it was just cloned
recursively (v.gr for modules with extenral dependencies)

Keep in mind that the containers created in this fashion are ephemeral,
so anything you write on the containers filesystem will be lost,
unless you remove the `--rm` option like so:

```bash
cd /path/to/project/root
docker run -it -h prezto --name prezto \
-v $(pwd):/home/prezto/.zprezto/modules/${:-$(pwd):t} \
hlecuanda/prezto-dev:latest
```
This will create a container named prezto, (`--name prezto`) with it's
hostname set to prezto also (`-h prezto`) that will retain changes
made to it's filesystem. When you detach, the container willi stop and
you can spinit up again using:

```bash
docker start -ai prezto
```

Containers started this way will remember the volume mounts they were
created with, so the project directory we previously mounted with
`-v` on the `docker run` command, will be ready on the image.

I have found epehermeral containers to be most useful since you get an
untainted, pristine environment for testing every time you spin up the
container.

Since the docker commands can be a bit verbose, the included Makefile
automates some of the frequent steps in the workflow, although it can
be replaced by [a shell function](https://gist.github.com/hlecuanda/78a6a39877c9753230a11c2d8832f4b6)
quite easily.


hope this turns out to be useful.
91 changes: 91 additions & 0 deletions prezto-container.zsh
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#!/usr/bin/env zsh
local opts withvals

zmodload zsh/zutil || { <<< 'Requires zparseopts'; false; return }

[ -z $commands[docker] ] && { <<< 'Requires Docker'; false; return }

zparseopts -D -E -M -a opts -A withvals - \
h=hlp -help=h \
i: -image=i \
N: -name=N \
n -dry-run=n \
p -persistant=p \
r -run=r \
-zdotdir: \
-zpreztodir:

if (( $#hlp == 1 )) ; then
<<-USAGE
${0}: create an ephemeral prezto container using docker

usage:
${0} [options] -- [addtl-args]

options:
-h, --help : print this message
-p, --persistant : make a persistant container
-N NAME, --name=NAME : set container name to NAME (default: prezto)
-n, --dry-run : see what command would be run without doing so
-r 'CMD' --run 'CMD' : run 'CMD' on the container (quote CMD)
-i IMG, --image=IMG : create container from image IMG (default hlecuanda/prezto-dev)
-zdotdir=PATH : use dotfiles from local PATH
-zpreztodir : override default prezto to local PATH

example:
${0} -n myruncoms -zdotdir=$HOME

creates an ephemeral container named myruncoms using
dotfiles found in ${HOME}

${0} -n illbeback -p

creates a persistant container named illbeback if such
container exists, then startit and attach to it

${0} -n ivebeenback -r 'apk add python'

spins up the a container named ivebeenback, and runs
the comand 'apk add python'. the container stops when
done.

USAGE
fi

local image="hlecuanda/prezto-dev"
local name="prezto"
local persistant="--rm"
local zdotdir=""
local zpreztodir=""
local dockercmd='docker run'
local dockerpull=""

for opt in ${(k)withvals}
case $opt in
-i)
image="$withvals[-i]" ;;
-n)
local dryrun="print --" ;;
-N)
name="$withvals[-n]" ;;
-p)
persistant="" ;;
-zdotdir)
zdotdir="-v ${(qq)withvals[-zdotdir]}:/home/prezto/zdotdir -e 'ZDOTDIR=/home/prezto/zdotdir " ;;
-zpreztodir)
zpreztodir="-v ${(qq)withvals[-zpreztodir]}:/home/prezto/zpreztodir -e 'ZPREZTODIR=/home/prezto/zpreztodir " ;;
esac

docker images \
| grep prezto-dev \
&>>! /dev/null || dockerpull="docker pull $image && "

dockercmd="$dryrun $dockerpull $dockercmd $persistant -h $name "
dockercmd="$dockercmd -name $name $zdotdir $zpreztodir $image"

cmd=$(echo $dockercmd | tr -s \ )

${(z)cmd}


# vim: set ft=zsh sw=2 tw=0 fdm=manual et :