Add CircleCI #5
Add CircleCI #5
Conversation
| @@ -46,7 +33,7 @@ RUN mkdir -p ${HOME}/lib/ ${HOME}/bin/ ${HOME}/data/models/ ${HOME}/src/ds-srv/ | |||
|
|
|||
| RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable | |||
|
|
|||
| RUN curl https://community-tc.services.mozilla.com/api/index/v1/task/project.deepspeech.deepspeech.native_client.v${DS_VER}.gpu/artifacts/public/native_client.tar.xz -sSL | pixz -d | tar -C ${HOME}/lib/ -xf - | |||
| RUN curl https://community-tc.services.mozilla.com/api/index/v1/task/project.deepspeech.deepspeech.native_client.v${DS_VER}.gpu/artifacts/public/native_client.tar.xz -sSL | xz -d | tar -C ${HOME}/lib/ -xf - | |||
|
|
|||
| RUN curl https://github.com/mozilla/DeepSpeech/releases/download/v${DS_VER}/deepspeech-${DS_VER}-models.tar.gz -sSL | gunzip | tar -C ${HOME}/data/models/ --strip-components 1 -xvf - | |||
There was a problem hiding this comment.
For now this is OK, but eventually we will want to have the ability to have custom models, one for Firefox Reality, one for Firefox Voice....
| @@ -46,7 +33,7 @@ RUN mkdir -p ${HOME}/lib/ ${HOME}/bin/ ${HOME}/data/models/ ${HOME}/src/ds-srv/ | |||
|
|
|||
| RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable | |||
|
|
|||
| RUN curl https://community-tc.services.mozilla.com/api/index/v1/task/project.deepspeech.deepspeech.native_client.v${DS_VER}.gpu/artifacts/public/native_client.tar.xz -sSL | pixz -d | tar -C ${HOME}/lib/ -xf - | |||
| RUN curl https://community-tc.services.mozilla.com/api/index/v1/task/project.deepspeech.deepspeech.native_client.v${DS_VER}.gpu/artifacts/public/native_client.tar.xz -sSL | xz -d | tar -C ${HOME}/lib/ -xf - | |||
There was a problem hiding this comment.
Not sure how much this is a problem ? But it looks good
There was a problem hiding this comment.
pixz doesn't support decompression to stdout, so this docker step wasn't putting these files in the docker image
There was a problem hiding this comment.
That would be unfortunate, since I'm using pixz -d to stdout in a lot of places on TaskCluster for DeepSpeech CI.
There was a problem hiding this comment.
$ ll lib/
total 0
drwxr-xr-x 1 alex alex 0 mars 4 18:06 .
drwxr-xr-x 1 alex alex 432 mars 4 18:06 ..
$ pixz --version
pixz: invalid option -- '-'
pixz: Parallel Indexing XZ compression, fully compatible with XZ
Basic usage:
pixz input output.pxz # Compress a file in parallel
pixz -d input.pxz output # Decompress
Tarballs:
pixz input.tar output.tpxz # Compress and index a tarball
pixz -d input.tpxz output.tar # Decompress
pixz -l input.tpxz # List tarball contents very fast
pixz -x path/to/file < input.tpxz | tar x # Extract one file very fast
tar -Ipixz -cf output.tpxz dir # Make tar use pixz automatically
Input and output:
pixz < input > output.pxz # Same as `pixz input output.pxz`
pixz -i input -o output.pxz # Ditto
pixz [-d] input # Automatically choose output filename
Other flags:
-0, -1 ... -9 Set compression level, from fastest to strongest
-p NUM Use a maximum of NUM CPU-intensive threads
-t Don't assume input is in tar format
-k Keep original input (do not remove it)
-c ignored -h Print this help
pixz 1.0.6
(C) 2009-2012 Dave Vasilevsky <[email protected]>
https://github.com/vasi/pixz
You may use this software under the FreeBSD License
$ curl https://community-tc.services.mozilla.com/api/index/v1/task/project.deepspeech.deepspeech.native_client.v0.6.1.gpu/artifacts/public/native_client.tar.xz -sSL | pixz -d | tar -C lib/ -xf -
can not seek in input: Illegal seek
alex@portable-alex:~/tmp/deepspeech/models$ ll lib/
total 72M
drwxr-xr-x 1 alex alex 144 mars 4 18:08 .
drwxr-xr-x 1 alex alex 432 mars 4 18:06 ..
-rwxr-xr-x 1 alex alex 892K janv. 10 18:22 deepspeech
-rw-r--r-- 1 alex alex 8,4K janv. 10 18:17 deepspeech.h
-r-xr-xr-x 1 alex alex 2,1M janv. 10 18:19 generate_trie
-r-xr-xr-x 1 alex alex 69M janv. 10 18:22 libdeepspeech.so
-rw-r--r-- 1 alex alex 17K janv. 10 18:17 LICENSE
-rw-r--r-- 1 alex alex 1,2K janv. 10 18:17 README.mozilla
There was a problem hiding this comment.
@jbuck So, indeed, it does support it. I'm worried that you see that kind of divergence.
There was a problem hiding this comment.
Ah, I was assuming it was a fatal error when I got this error while running the docker build:
Step 6/7 : RUN curl https://community-tc.services.mozilla.com/api/index/v1/task/project.deepspeech.deepspeech.native_client.v${DS_VER}.gpu/artifacts/public/native_client.tar.xz -sSL | pixz -d | tar -C ${HOME}/lib/ -xf -
---> Running in 4e8e178a20ad
can not seek in input: Illegal seek
Removing intermediate container 4e8e178a20ad
---> 1e097ede4bd9
But it did decompress the files successfully inside the docker image
It looks like this is filed as a bug on the pixz repo: vasi/pixz#62
No description provided.