Introduce Grunt-supporting Node Template

[jayharris]

The current node bash and batch scripts both run npm install --production within wwwroot. However, sometimes there are tasks that need to be executed as --development, such as Grunt, where we need to compile LESS or Sass files into CSS, compile CoffeeScript or TypeScript into JS, or optimization tasks such as bundling and minification.

With Grunt becoming such commonplace within the node ecosystem, it would be helpful if there was a KuduScript template that could execute tasks before the KuduSync from SOURCE to TARGET.

@davidebbo suggested opening a ticket here to begin discussions on this idea.

Possiblities & Considerations

Run Grunt

Similar to the check for package.json before running npm install, we can check for the presence of a Gruntfile (which can be either Gruntfile.js or Gruntfile.coffee). If the file is found, npm install grunt-cli and execute, similar to:

if [ -e "$DEPLOYMENT_SOURCE/Gruntfile.js" ]; then  
  eval $NPM_CMD install grunt-cli  
  exitWithMessageOnError "installing grunt failed"  
  ./node_modules/.bin/grunt --no-color clean common dist  
  exitWithMessageOnError "grunt failed"  
fi

Though grunt is commonly a package.json, grunt-cli is needed to get the grunt command line tool into the local node_modules folder, so the npm install grunt-cli is necessary.

Perhaps this template could be available via a --grunt TASKNAME command line argument, with the task name value injected in place of clean common dist above.

Run Bower

Similar to checking for Grunt, we could check for the existence of bower.json, and if it exists, npm install bower and ./node_modules/.bin/bower install.

if [ -e "$DEPLOYMENT_SOURCE/bower.json" ]; then  
  eval $NPM_CMD install bower  
  exitWithMessageOnError "installing bower failed"  
  ./node_modules/.bin/bower install  
  exitWithMessageOnError "bower failed"  
fi

This step should execute before grunt.

Alternately, we could direct developers to use grunt-bower-task, a Grunt task that executes bower install. This way, we can combine everything in to the grunt, and not have additional tooling.

App path and Distribution path

Similar to how an aspWAP may have a solution file/folder and a project file/folder, we may need similar capabilities for this template. If the project is not in the repository root (perhaps their package.json is in the ./src folder), then there needs to be a flag to specify that path. Additionally Grunt is regularly used to generate a production-ready folder, such as ./dist, which has all of the compiled and optimized folders, so we will need another flag for that.

Personally, I think the former is a rare case, and goes against a lot of the convention of node projects, and in particular, Grunt-based projects. However, the latter is very common, especially with frameworks such as Lineman and Yeoman.

If we just went with the latter ./dist option, this could be accomplished through --sitePath.

[davidebbo]

Does this need to be an entirely new template, or is this more about teaching new tricks to the existing Node template? Could we say that the Bower step always happens if bower.json is detected, such that it would not even require additional flags to trigger that? I guess there is a small chance that it would break some existing projects that don't expect this to happen even though they have that file?

In any case, you understand those Grunt/Bower workflows way better than we do, so anything you can help with and contribute to the templates would be welcome!

[jayharris]

One concern with just augmenting the current Node script is that there seems to be two very different Node scenarios.

Production Dependencies

The current Node template immediately syncs the Repo to wwwroot, and once it is synced, then it will npm install to pull in the Production dependencies. bower install may be needed here, such as if the site's HTML is directly referencing jQuery from the bower_components folder. npm install is needed here to pull in things like express.

Development Dependencies

An aspWAP project will need to be compiled in the Repo / Source folder, and the compiled output KuduSynced from Source to Target. Node will have similar scenario, where Pre-sync tasks need to occur. Bower may be necessary here as well, such as if jQuery is used in the site, but not referenced directly because it has been bundled with the rest of the site's JavaScript into a single app.js. npm install is needed here as well, to pull in things like Grunt.

Of course, there's also a "Both" option, if there is Preprocessing that needs to happen, such as bundling and minification, but the site is still dynamic at runtime, such as if it is a blog or has SendGrid integration. This is the scenario that I'm really struggling to wrap my head around for a template implementation. Because this is very possible and likely, a single script does seem to make sense, but what does that look like?

1. npm install
2. bower install
3. grunt install
4. KuduSync the generated ./dist folder from repo/dist to wwwroot
5. npm install, again (because the ./dist outcome could have it's own set of dependencies)

It could work. We'd just need to add 1-3 to the existing template.

[ambroselittle]

I more or less concur with @jayharris. As a rule, I'd be more concerned with complicating/adding options to a specific generator. I'd think it is easier to discover, understand, and use, if you think more in terms of more specific workflows.

It could be a "grunt-build" workflow generator that maps to what we are talking about here, which is a pretty common one and is, e.g., used by popular Yeoman generators. I think they use "app" for the "dist" folder from what I recall off hand.

The flow is basically what @jayharris outlines above. Points of configuration are:

• whether or not bower is used (normally it is, but you could detect, as you say)
• which grunt task to use - for this, you could just define one for azure, like "grunt azure" and let people just add a task to their grunt files. I think it is going to be easier/more straightforward for most to change their gruntfile than deploy.sh, just for familiarity's sake.

[jayharris]

I agree. The only additional thought I would have is to accept an additional value on the grunt flag. --grunt TASKNAME, so that it wouldn't be fixed to azure as a task name. If my task is build, I can use --grunt build, rather than having to modify Gruntfile to add an azure task simply for wrapping build. The template would handle this for me, so I wouldn't have to modify deploy.sh, either.

[davidebbo]

As an aside, I want to point out that there are two very different workflows for using these templates (generally, not specific to Node):

1. You run the azure tool locally and generate them. At this point, you have a 'custom' script, which you can modify as you like.
2. Directly use a template that gets generated by Kudu on the server for you. Kudu has some default heuristic to determine how to launch the generator, but you can override it all by using the SCM_SCRIPT_GENERATOR_ARGS setting (see this page for details).

Technically, any improvement you make to the template generation would apply to both, but I'm curious if you're primarily interested in 1 or on 2. In other words, do you think the generator can produce something that directly does everything you need (given the right set of params), or is it more of a starting point to customize further?

[jayharris]

I absolutely see this as something that applies to both scenarios.
For many cases, --grunt build and you are done; commit and push. So there's no reason why we couldn't use that as our flag in SCM_SCRIPT_GENERATOR_ARGS.

Also, related to @ambroselittle, I some additional research and build seems to be the most common convention for a grunt task that does the production preparation. Perhaps we could assume build is the task (rather than make an azure task) that we execute if you do not specify a task name.

We could use the build task assumption for heuristics. Fork NodeSiteBuilder, perhaps into a GruntSiteBuilder, and choose which in the ResolveNonAspProject factory function based on the existence of a Gruntfile.

[ambroselittle]

That sounds fine to me.

BTW, I find I need to run npm install for each of my submodules.

Will adding this to my deploy.sh after the parent npm install work?

eval "git submodule foreach npm install"

[ambroselittle]

Turns out this seems to work for that eval git submodule foreach npm install, surprisingly. I guess npm is in the right place on the server and doesn't need the full path.

[davidebbo]

If you push that direction, wouldn't the bower/grunt/etc logic also need to be done for each git submodule?

[ambroselittle]

Yes, well... that's the prob I'm in now, because I use grunt-shell to run grunt build on my submodules in my Gruntfile. :/

BTW, I do just specify bower as a dependency in package.json, and have npm install run bower install, and that works for me. I did a little research, and there's no good/right way to specify a global dependency. It doesn't seem ideal to me to rely on global installs..

I'm just about to try tweaking my build script to run the local grunt from grunt shell to see how that works..

[ambroselittle]

Well grunt-shell doesn't seem to work. I guess next option is to customize my deploy script to call build on each submodule and then customize my grunt file to not try to use shell in this context.

But I'm done for now. See y'all Monday. :)

[davidebbo]

Is it fair to say that to some extent, this goes beyond Azure Web Sites? i.e. if you have a repo with various submodules, each of which needing a number of build tasks, then presumably you would need some easy workflow to perform all these tasks from any enlistment. Once that workflow exists, a Kudu script can just hook into that rather than having to do it all itself.

[jayharris]

If you are using grunt-shell to run grunt on Azure, it will fail because the executed grunt command isn't globally installed. Just as we would need to in our Kudu script, you will need to install grunt-cli and then run ./node_modules/.bin/grunt. Also, with the eval "git submodule foreach npm install" command, it may be better off running eval "git submodule foreach $NPM_CMD install", so that npm is running underneath the currently selected version of node. Otherwise, you may run in to conflicts if your package.json references a different version of node than what is installed by default on the WAWS server.

[ambroselittle]

I tried those. It wasn't working.

from my tablet

On Dec 6, 2013, at 8:55 PM, Jay Harris [email protected] wrote:

If you are using grunt-shell to run grunt on Azure, it will fail because the executed grunt command isn't globally installed. Just as we would need to in our Kudu script, you will need to install grunt-cli and then run ./node_modules/.bin/grunt. Also, with the eval "git submodule foreach npm install" command, it may be better off running eval "git submodule foreach $NPM_CMD install", so that npm is running underneath the currently selected version of node. Otherwise, you may run in to conflicts if your package.json references a different version of node than what is installed by default on the WAWS server.

—
Reply to this email directly or view it on GitHub.

[ambroselittle]

@davidebbo I bet you could argue either way. Certainly submodules are less common than not having them. That said, if you had a basic setup that worked with them, it would help, of course. Or at least a reference/sample people could follow.

You could check for .gitmodules and use that to toggle the logic on whether or not to try npm install, bower, & grunt build on them.

[ambroselittle]

Okay, guys, I finally got it working. Mostly.

I created custom grunt tasks that omit the call to grunt-shell and instead run this:
git submodule foreach ./node_modules/.bin/grunt --no-color --force build

The --force is needed because I am using grunt-contrib-clean, and I guess it runs in the .bin directory context instead of the project root..

I also tweaked the script to run using the selected node version:
git submodule foreach "$NPM_CMD" install

It was erroring out before because I was using eval, I guess.

BTW, for some reason the script takes an unreasonable amount of time to process. I run the build locally in a few seconds. It takes about 5 minutes to complete deployment on the server. If it makes it. I've gotten this error twice now:

Command 'starter.cmd bash deploy.sh' aborted due to no output and CPU activity for 60 seconds.  
You may increase SCM_COMMAND_IDLE_TIMEOUT setting to solve the issue.
starter.cmd bash deploy.sh

[davidebbo]

Usually, that happens when something is hanging, e.g. prompting for input. It monitors CPU activity, so it only kills things when nothing at all seems to be happening.

[ambroselittle]

Weird that it seems to be intermittent, then. I'd expect something like that to happen every time.

[davidebbo]

Yes, I agree, that is weird. If you have a repro for that, we will investigate. Also, you may be able to get some info by hitting the 'Processes' link from the root of the Kudu service. This REST API will show you the list of running processes, as well as their memory/cpu counters.

[ambroselittle]

Well, here's my deploy script, FWIW:

##################################################################################################################################
# Deployment
# ----------

echo Handling node.js with grunt build deployment.

# 1. Select node version
selectNodeVersion

# 2. Install npm packages 
if [ -e "$DEPLOYMENT_SOURCE/package.json" ]; then
  eval $NPM_CMD install
  exitWithMessageOnError "npm failed"
  git submodule foreach "$NPM_CMD" install
fi

# 3. Run grunt to build
if [ -e "$DEPLOYMENT_SOURCE/Gruntfile.js" ]; then  
  eval $NPM_CMD install grunt-cli  
  exitWithMessageOnError "installing grunt failed"  
  ./node_modules/.bin/grunt --no-color --force cleanModules 
  git submodule foreach ./node_modules/.bin/grunt --no-color --force build
  ./node_modules/.bin/grunt --no-color --force dist 
  exitWithMessageOnError "grunt failed"  
fi  

# 4. KuduSync
"$KUDU_SYNC_CMD" -v 50 -f "$DEPLOYMENT_SOURCE/dist" -t "$DEPLOYMENT_TARGET" -n "$NEXT_MANIFEST_PATH" -p "$PREVIOUS_MANIFEST_PATH" -i ".git;.hg;.deployment;deploy.sh"
exitWithMessageOnError "Kudu Sync failed"


##################################################################################################################################

[ambroselittle]

Could it be just the sheer number of files? I have a lot cuz I haven't put build optimization in place yet (and some of the stuff just has a lot of files anyways by its nature). Maybe my CPU time is being throttled enough that sometimes the copying operations just time out.

Also, because I follow the clean and build approach, all files are always shown as modified, so they all always get sync'd. I was looking at grunt-sync, but it still doesn't deal with removing/dealing with old/orphan stuff that changes during dev, so the clean-build approach seems better. Not sure I see a way around that, just mentioning it for your info..

[davidebbo]

If you have a test repo that I can use, I can try deploying it to a Standard (reserved) instance, where there is no CPU limit. That would allow us to confirm if this is the cause.

You could also check your CPU meter on the Azure portal to see if it rises significantly as a result of this operation.

[ambroselittle]

I don't have any repro I could create in a reasonable amount of time.

I did just confirm, running the commands in the console, that one of the commands takes almost two minutes to complete. It's just copying a bunch of directories and files. Takes a couple seconds on my box, but two minutes on the server. And the CPU time shoots up in the dashboard, too.

So I guess that's what it is.. any way to set the timeout higher (as a quick fix until I can get back to optimizing it somehow)?

[davidebbo]

Yes, and the error message tells you exactly how! :) Just set SCM_COMMAND_IDLE_TIMEOUT to a value higher than 60. This would be a good test.

But something doesn't quite add up here, as the timeout should not kick in if there is CPU activity. Conceivably that logic may have bugs. Anyway, try some high number and see what happens.

[ambroselittle]

You caught me. :) I forgot it said that.

Well, anyways, it looks like I have a new issue with the jsdom dependency. I am gonna have to put further work on this on hold. Thanks for all your help!

[jayharris]

Should sub modules be included in the stock Grunt template? What if the base repo is Hg?

Perhaps we should open a new ticked for remote template support, similar to Heroku Buildpacks, and cover sub modules under that scenario?

[davidebbo]

Good question. Today, all of the deployment templates are agnostic of source control. The general assumption is that by the time the template is called, all the files are in the correct source control state. Note that there is logic in Kudu to update the submodules before calling the deployment template, though in some cases it takes some slightly tricky steps to get it right (due to auth).

Though in this case, maybe you're not talking about the logic that updates the submodule sources, but more about logic that does further grunt processing on them. Correct?

[jayharris]

Correct. Kudu logic will pull latest on each submodule before we get to deploy.sh, but with the commands from @ambroselittle, we are still executing git submodule foreach, so we would still have a hard dependency on git.

Though I think this is very valuable, I think it would break the spirit of the default templates.

(Also where I think there would be value of remote/custom templates.)

[jayharris]

I'm planning on tackling this shortly, implementing this into the base templates and preparing a PR. I just have to wait until after CodeMash, this week.

Is this still something we think would be valuable?

[ambroselittle]

I definitely think it is useful for the ecosystem to be broadened for AWS.

If the goal is independence from any SCC, then I'd say it is safe to remove the submodule init. In which case, it would be helpful to point folks to an example or something if they have submodules. OTOH, if Git is the 80% case, it may be more helpful to assume it and then provide directions for those not using it. Besides, I guess the command itself shouldn't block overall success if there are no submodules.

[davidebbo]

What specific approach are you planning to go with? From out point of view, as long as it doesn't hurt other scenarios and adds value for some users, it's a good thing.

[stereokai]

I don't know if this discussion is still relevant (I haven't been following, just got into Azure Websites in the past 2 days due to my new job), but perhaps a tool like wiredep (or grunt-wiredep) can be of help?