Automatically Build NPM Module On Install From Github

Given that a project's lib/ dir shouldn't be checked into Git because the files it contains are derived files (from the build process). When installing a package from the project's github (during development for example) the lib/ dir will not exist, so if the package's package.json's main field points to (for example) lib/index.js, the package cannot be compiled when imported because these files do not exist within the repository and therefore within the package installed into node_modules. This means the package needs to built (just as it would be before release), only this time locally so that the lib directory (or whatever other files are generated during the build process) are added to the module's directory.

Assuming there is a build script within the package.json file's scripts field, can the package be configured to run this automatically in the situation where it is installed from github only? If not, what is the the best approach to ensuring it is built when installed from github?

There are now prepublish, prepublishOnly and prepare lifecycle hooks, but none provide an answer to this problem because they don't allow any way to differentiate between the source of the install. In short, yes they allow you to build on install, but they don't allow you to build on install from github only. There is not only no reason to force a build when people install from npm, but more importantly, development dependencies will not be installed (for example babel which is critical to the build).

I am aware of one strategy to deal with this problem:

  • Fork / branch the repo
  • build locally
  • remove the lib/ dir from .gitignore and check it in.
  • install module from your fork/branch
  • When you are ready for a PR / rebase add lib/ dir to .gitignore and remove dir from git.

But that is far from ideal. I guess this could be automated with a githook though. So every you push to master the project also builds and pushes to a separate branch.

There is a closed issue on NPM's Github with no resolution - just lots of people who want a solution. From this issue it is clear that using prepare is not the answer.

My usecase is that I am developing a module that is used in a number of other projects. I want to consume the latest version of the module without having to push out a release to NPM whenever I update the codebase - I would rather push out fewer releases when I am ready, but I still need to consume whatever the latest version of the lib is on Github.

Note: I also reached out to NPM's support regarding this issue and I'll add their response if I receive one.

24877 次浏览
小开

Edit: Detecting if the package is being installed from git repo

I didn't understand the question properly. Below are things that I wrote but are a bit off-topic. For now if you want to run build.js only when installing from repo:

Files in repo:

 .gitignore
.npmignore
ThisIsNpmPackage
build.js
package.json

The .gitginore:

ThisIsNpmPackage

The .npmignore:

!ThisIsNpmPackage

In the package.json:

"scripts": {
"install": "( [ ! -f ./ThisIsNpmPackage ] && [ ! -f ./AlreadyInstalled ] && echo \"\" > ./AlreadyInstalled && npm install . && node ./build.js ) || echo \"SKIP: NON GIT SOURCE\""
}

The idea is to make file ThisIsNpmPackage available on the repo, but not in npm package.

Install hook it's just a piece of bashy script to check if ThisIsNpmPackage exists. If yes then we execute npm install . (this will ensure we have devDependencies. File AlreadyInstalled is generated to prevent infinite looping (npm install will recursively invoke install hook)

When publishing I do git push and npm publish
Note that npm publish can be automated via CI tools - githooks

This little hack with file ThisIsNpmPackage makes the source detection available.

Results of invoking npm install dumb-package:

"SKIP: NON-GIT SOURCE"

And executing npm install https://github.com/styczynski/dumb-package

Files will be built

The issues

The main issues we are facing here are the following ones:

  • Have to do npm publish ... everytime

    Sometimes it's too much pain to fix a small bug, then push to the repo and forget to publish on npm. When I was working with a microservices-based project that has about 5 standalone subprojects divided into modules the problem that I found an issue, fixed it and forget to publish in every place I had to was really annoying.

  • Don't want to push lib into the repo, because it's derived from sources

  • Rebasing and merging is even more annoying.

  • No mess with .gitgnore

    Heck, I know that problem when you have a troublesome files that you have to include inside repo but never modify them, or sometimes remove? That's just sick.

Edit: npm hooks

As @Roy Tinker mentioned there exist ability for a package to execute a command when installed.
It can be achieved via npm hooks.

"install": "npm run build"

And we execute the:

npm install https://github.com/<user>/<package>

Edit:
OP question from comments:

But this will run an install for everyone downloading the module from npm right? This is hugely problematic given that dev dependencies will not be installed for anyone downloading the module from npm. The libs used to build the app - babel etc will not be installed.

Note: But if you want a specific version of the package (production/dev) with or without dev dependencies you can install it via:

npm install --only=dev

The --only={prod[uction]|dev[elopment]} argument will cause either only devDependencies or only non-devDependencies to be installed regardless of the NODE_ENV.

A better solution, in my opinion, is to use:

npm install <git remote url>

And then inside package.json specify:

"scripts": {
"prepare": "npm run build"
}

If the package being installed contains a prepare script, its dependencies and devDependencies will be installed, and the prepare script will be run, before the package is packaged and installed.

Example:

npm install git+https://isaacs@github.com/npm/npm.git

Read the npm docs there: npm install

Edit: proxy module (advanced technique)

It's kind of bad practice, but good to know.

Sometimes (as in case of Electron framework you need to install other external packages or resources/modules depending on various conditions).

In these cases the proxy idea is used:

  • You make a module that behaves like installer and installs all depending things you want

In your case prepare script will be enough, but I leave this option, because it may be sometimes helpful.

The idea is that you write a module and write a install kook for it:

"scripts": {
"install": "<do the install>"
}

In this scenario you can place there:

npm install . && npm run build

Which install all devDependencies anyway (as beforementioned prepare case do), but it's a bit of hacking.

If you want do the real hacking there:

 "scripts": {
"install": "curl -L -J -O \"<some_url>\""
}

which manually download files using -nix command curl

It should be avoided but it's an option in case of the module that has huge binary files for each platform and you don't want to install them all.

Like in case of Electron where you have compiled binaries (each for the separate platform)

So you want people to make install package not install package-linux or package-window etc.

So you provide custom install script in the package.json

{
...
"scripts": {
"install": "node ./install_platform_dep.js"
}
}

Then when installing module the install_platform_dep.js script will be executed. Inside install_platform_dep.js you place:

// For Windows...
if(process.platform === 'win32') {
// Trigger Windows module installation
exec('npm install fancy-module-windows', (err, stdout, stderr) => {
// Some error handling...
}
} else if ... // Process other OS'es

And this in purely manual way installs everything.

Note: Once again this approach is usable with platform-depending modules and if you use that it's probably design issue with your code.

Build on CI

What comes to my mind is the solution that I used really for a long time (automatic building with CI services).

Most of the CI services' main purpose is to test/build/publish your code when pushing to the branch or doing other actions with the repo.

The idea is that you provide settings file (like travis.yml or .gitlab-ci.yml) and the tools take care of the rest.

If you really don't want to include the lib into the project, just trust CI to do everything:

  • Githook将在提交时触发构建（在分支或任何其他分支上-这只是配置的问题）
  • CI将构建您的文件，然后将其传递到测试阶段并发布

现在我在我自己的项目上做吉特实验室（作为业余爱好的一部分）一些网页。构建项目的吉特实验室配置如下所示:

image: tetraweb/php


cache:
untracked: true
paths:
- public_html/
- node_modules/


before_script:
- apt-get update


stages:
- build
- test
- deploy
  

build_product:
stage: build
script:
- npm run test


build_product:
stage: build
script:
- npm run build
  

deploy_product:
stage: deploy
script:
- npm run deploy

当我合并到主分支时，会发生以下事件:

  • CI运行build阶段
  • 如果构建成功，则启动test阶段
  • 如果test阶段正常，则最终触发阶段deploy

脚本是要执行的UNIX命令的列表。

您可以在配置中指定任何Docker映像，因此实际上可以使用您想要的任何UNIX版本以及一些（或不）预安装的工具。

有一个软件包部署到Git，它将人工制品部署到所需的回购分支。

或者在这里（对于Travis CI），将人工制品发布到repo的配置:

Travis-发布到Git

（我自己用的）

然后，当然可以让CI运行:

npm publish .

因为CI执行UNIX命令，所以它可以（至少有一组CI提供程序）:

  • 发布标签（可能是发布标签？）
  • 触发器脚本在所有自述文件和任何地方更新项目的版本
  • 如果所有阶段都成功，则向您发送通知

So what I do:
I commit, push and let the tools do everything else I want.
In the meantime, I make other changes and after one to ten minutes get update report by mail.

There is plenty of CI provider there:

Here I attach another example of my other project (.travis.yml):

language: generic
install:
- npm install
script:
- chmod u+x ./utest.sh
- chmod u+x ./self_test/autodetection_cli/someprogram.sh
- cd self_test && bash ../utest.sh --ttools stime --tno-spinner

If you set up CI to push and publish your package you can be always sure to use the latest cutting-edge version of your code without worrying about eh I have to run also this command now... problem.

I recommend you to choose one of the CI providers out there.
The best ones offer you hundreds of abilities!

When you get used to automatically doing publish, test and build phases you will see how it helps to enjoy the life!
Then to start another project with automatic scripts just copy the configs!

Summary

In my opinion npm prepare script is an option.
You can also maybe want to try others.

Each of the described methods has it's drawbacks and can be used depending on what you want to achieve.
I just want to provide some alternatives hope some of them will fit your problem!

小开

Assuming there is a build script within the package.json file's scripts field, can the package be configured to run this automatically in this situation?

Yes. There are 2 things you need to do:

  1. Make sure your system uses npm or yarn to install the package from GitHub. If this package is a dependency of another package, you can use the GitHub URL in place of the version number in package.json. Otherwise, the following command will work:

    npm install https://github.com/YourUser/your-package

    You can add /tags/v1.0.0 or whatever to the end of the URL if you're after a specific tag or branch.

  2. Add the following to the scripts in your module's package.json:

    "install": "npm run build"

install is a hook that the package manager executes after installing the module. (preinstall and postinstall also -- see documentation).

Documentation: https://docs.npmjs.com/misc/scripts

小开

EDITED 2

It's a great question. It's too bad there isn't a recognized reliable solution, but the following seems to work.

Create a .buildme marker file, and commit to git.

In package.json:

  "files": ["lib"],
"scripts": {
"build": "echo DO WHAT YOU NEED TO BUILD",
"prepack": "[ ! -f .buildme ] || npm run build",
"preinstall": "[ ! -f .buildme ] || npm run build"
},

Here are the things to note.

  1. The special .buildme marker file should be excluded from the npm package with either the "files" key, or via .npmignore.

  2. The prepack hook runs when you publish (prepublishOnly might also work, but it's nice that with prepack, npm pack will produce a correct tarball).

  3. When installing from npm, preinstall runs, but does nothing because .buildme is missing (thanks to the the [ ! -f .buildme ] clause).

  4. When installing from github, .buildme does exist. On npm6, prepack hook runs the build (and produces a package without .buildme), and preinstall does nothing. On yarn 1.12, preinstall does the build.

  5. If you install an updated version from github, preinstall will run again, and will build again.

NOTE: When installing from github, it's up to the person installing to have enough of your package's devDependencies already installed for the build to work. (This solution doesn't attempt to auto-install devDependencies.)

That's it. It seems to work with varios combinations of npm 6 and yarn 1.12.

小开

prepare is the correct way, but might seem broken

If you have a repository with source files but a "build" step is necessary to use it,
prepare does exactly what you want in all cases (as of npm 4).

prepare: Run both BEFORE the package is packed and published, on local npm install without any arguments, and when installing git dependencies.

You can even put your build dependencies into devDependencies and they will be installed before prepare is executed.

Here is an example of a package of mine that uses this method.

Problems with .gitignore - prepare will seem broken

There is one issue with this option that gets many people. When preparing a dependency, Npm and Yarn will keep only the files that are listed in the files section of package.json.

One might see that files defaults to all files being included and think they're done. What is easily missed is that:

  • .npmignore mostly overrides the files directive and,
  • if .npmignore does not exist, .gitignore is used instead.

So, if you have your built files listed in .gitignore, like a sane person, and don't do anything else, prepare will seem broken.

If you fix files to only include the built files or add an empty .npmignore, you're all set.

My recommendation is to set files (or, by inversion, .npmignore) such that the only files actually published are those needed by users of the published package. Imho, there is no need to include uncompiled sources in published packages.


提交答案