Modularity
From thecentric
Contents |
Introduction
modularity is a set of Ant tasks to help with the software configuration management of Java projects with a large number of modules. modularity aims to make maintaining ant build files a thing of the past - instead you maintain a set of metadata xml files describing your project's and module's layout and dependency tree.
This project builds on the work originally carried out by Malcolm Sparks.
Currently modularity, in its form described below, is an incubator project at NewBay Software who use modularity for all their engineering projects.
Features
- Keep all dependency information in one place. (Instead of one set of dependency information embedded in your IDE and another in your build).
- Easily add or extract modules
- No overhead in hacking or maintaining build.xml files.
- No overhead on a multi person team in sharing out IDE configurations.
- Ant + Freemarker - uses standard build and templating engines.
- If you're an Ant or Freemarker ninja - you're already a modularity ninja.
- Model Extendable - All information added to the module metadata can be used in the template.
- No conventions imposed.
- Write templates that reflect custom SCM processes and procedures.
- In a multi project setups - reuse the templates for quick bootstrapping and consistant approach.
- Plugs into all existing Continous Integration servers.
- Transistive Dependency Support:
- inherit none
- inherit all
- inherit exported
Terminology
- 3rd Party Package Release This is a package distributed by another project - like hibernate-3.2.4 for example.
- Global Package Cache - The GPC is a repository of 3rd party package releases. Packages like hibernate-3.2.4 and commons-collections-3.2.
- Module - A module is a discrete functional unit of software that has ideally a single purpose. A module will usually have sources, tests and resources and have a described set of dependencies on other modules inside a project or on 3rd party package releases.
- Module Group - A module group is a collection of modules sharing a common version history. A module group is the lowest unit of versioning. Each module in a module group shares the same lifecycle.
- Metadata - Information about a module's set of dependencies as well as any other information about the module that may be useful in the build process. It is described in xml usually in a file called metadata.xml located in the root of the module.
- Dependency Graph - A directed acyclic graph - i.e. a dependency tree that contains no cyclical dependencies.
The Global Package Cache
|
The GPC is a global repository of 3rd party package releases. Packages like hibernate-3.2.4 and commons-collections-3.2.
It is recommeneded that the GPC contain a complete set of files assocated with a release of the package - including jars, libraries, source, javadoc, tutorials, examples etc...
As a rule of thumb the GPC should only ever be added to - files should never need to be modified or removed.
An example file system layout of the GPC is shown below with only two packages:
- gpc
|
+ hibernate-3.2.4
|
- logging-log4j-1.2.13
|
+ dist
|
+ docs
|
+ examples
|
+ src
The GPC is described by an xml file - usually called gpc.xml. This file contains the following information about the package release:
- directory location
- package name (Example: logging-log4j)
- package version (Example: 1.2.13)
- jar locations
- package dependencies
- javadoc location
- source location
An example of a gpc.xml file is given below with two mock entries:
<gpc>
<package name="log4j" version="1.2.13" dir="logging-log4j-1.2.13">
<jar location="dist/lib/log4j-1.2.13.jar"/>
<source location="src"/>
<javadoc location="docs/api"/>
</package>
<package name="hibernate" version="3.2.4" dir="hibernate-3.2.4">
...
<dependencies>
<package name="log4j" version="1.2.13">
</dependencies>
</package>
</gpc>
The GPC description file must be hand maintained. The required attributes of package are 'name', 'version' and 'dir'. Usually for a package entry to become useful at least a single jar element is needed.
Metadata
|
Information about a module / module group is stored in a file called metadata.xml in the root of the module or module group. There is a small number of required elements and attributes in the metadata, the rest you will be defining most of it yourself as necessary.
Adding information into the metadata makes it available to ant templates later in the modular build process.
Information that may typically stored be stored in a modules metadata include:
- name
- type (for example java/web/j2me)
- module dependencies
- package dependencies
- source location
- resource location (file to go into classpath)
- web resource locations
- release configurations
A modulegroup's metadata usually contains the following:
- current version
- list of modules in module group
- dependency profile
Let's take a hypothetical web project called petstore with two module groups, and each module group contains two modules:
- petstore
- petstore-web
- petstore-core
- common-utilities
- string-utilities
- hibernate-utilities
black - elements and attributes defind by modularity
blue - custom elements and attributes
module group petstore metadata.xml
since we are importing all package and module dependencies here, we only need to define
one dependency 'petstore-web' - which is at the top of the dependency graph.
<moduleGroup name="petstore" version="2.1.3-DEV">
<dependencies importAllPackages="true" importAllModules="true">
<module name="petstore-web"/>
</dependencies>
<dependencyProfile>
<packageConflict name="log4j" useReleaseName="log4j-1.2.9"/>
</dependencyProfile>
</moduleGroup>
module petstore-web metadata.xml
<module name="petstore-web" type="web">
<dependencies>
<package releaseName="commons-lang-2.1"/>
<package releaseName="hibernate-3.2.4"/>
<package releaseName="log4j-1.2.9/>
<module name="petstore-core"/>
<moduleGroup name="common-utilities>
<module name="file-utilities"/>
<module name="hibernate-utilities"/>
</moduleGroup>
</dependencies>
<source location="src/java">
<testSource location="test/java">
<resource location="src/resources">
<webapp location="web">
<release name="default" directory="releases/default">
<resource location="release-configurations/default">
</release>
</module>
module petstore-core metadata.xml
<module name="petstore-core" type="java">
<dependencies importAllPackages="true">
<moduleGroup name="common-utilities>
<module name="file-utilities"/>
<module name="hibernate-utilities"/>
</moduleGroup>
</dependencies>
<source location="src/java">
</moduleGroup>
module hibernate-utilities metadata.xml
<module name="hibernate-utilities" type="java">
<dependencies>
<package releaseName="hibernate-3.2.4"/>
<package releaseName="log4j-1.2.13/>
</dependencies>
<source location="src/java">
</moduleGroup>
module file-utilities metadata.xml
<module name="file-utilities" type="java">
<dependencies>
<package releaseName="commons-lang-2.1"/>
<package releaseName="log4j-1.2.13/>
</dependencies>
<source location="src/java">
</moduleGroup>
Templates
The following is a representation of the model made available to the template by default with modularity. This model can be extended by adding your own information into a modules metadata.xml or the GPC's gpc.xml.
(root)
|
+- buildDir -> STRING: buildDir attribute of m:build
|
+- (parameter) -> MAP: the parameters passed to m:execute
|
+- (gpc) -> MAP
|
+- dir -> STRING: gpcDir attribute of m:build
|
+- ... ->
| -> further entries populated from gpc.xml
+- ... ->
|
+- (module) -> MAP :
|
+- dir -> STRING: the modules directory
|
+- version -> STRING: version -inherited from the module group
|
+- buildDir -> STRING: helper delegate for modules scatch build
| directory - same as:
| ${buildDir}/modules/${module.name}
|
+- buildJar -> STRING: helper delegate for this module's
| resulting jar - same as:
| ${module.buildDir}/
| ${module.name}-${module.version}.jar
|
+- moduleDependencies -> LIST: of module dependencies described below
|
+- (moduleDependency) -> MAP
|
+- ... -> same map as the module dependency's
| -> original map. further entries
| -> are also populated from attributes in the
+- ... -> dependency element of the metadata.xml
|
+- packageDependencies -> LIST: of package dependencies described below
|
+- (packageDependency) -> MAP
|
+- dir -> STRING: the packages directory
|
+- name -> STRING: the packages name
|
+- releaseName -> STRING: the packages release name
|
+- jars -> LIST: of jar dependencies described below
|
+- (jar) -> MAP:
|
+- filepath -> STRING: full path to one of the package
| -> dependencies jar
|
+- filename -> STRING: the package dependencies jar's filename
+- ... ->
| -> further entries are populated from
| -> attributes in the
+- ... -> dependency element of the metadata.xml
|
+- ... ->
| -> further entries populated from the
| -> module's metadata.xml
+- ... ->
Here is an trivial example of a modularity template - that echoes some different values in the model.
I've mixed and matched different ways of calling through to the echo's for demonstration purposes:
blue - freemarker
<project name="${module.name}.echo" default="echo" basedir="${module.dir}">
<target name="echo.package.dependencies">
<#list module.packageDependencies as packageDependency>
<echo message="package: ${packageDependency.releaseName}"/>
</#list>
</target>
<#list module.moduleDependencies as moduleDependency>
<target name="${moduleDependency.name}.echo">
<echo message="module: ${moduleDependency.name}"/>
</target>
</#list>
<target name="echo" depends="echo.package.dependencies">
<echo message="version: ${module.version}"/>
<echo message="gpc directory: ${gpc.dir}"/>
<#list module.moduleDependencies as moduleDependency>
<antcall target="${moduleDependency.name}.echo"/>
</#list>
</target>
</project>
Now imagine we passed the module 'petstore-core' from the previous section through this template - the result would be this:
<project name="petstore-core.echo" default="echo"
basedir="/projects/petstore/petstore-core">
<target name="echo.package.dependencies">
<echo message="package: commons-lang-2.1"/>
<echo message="package: hibernate-3.2.4/>
<echo message="package: log4j-1.2.9"/>
</target>
<target name="file-utilities.echo">
<echo message="module: file-utilities"/>
</target>
<target name="hibernate-utilities.echo">
<echo message="module: hibernate-utilities"/>
</target>
<target name="echo" depends="echo.package.dependencies">
<echo message="version: 2.1.3-DEV"/>
<echo message="gpc directory: /projects/gpc"/>
<antcall target="file-utilities.echo"/>
<antcall target="hibernate-utilities.echo"/>
</target>
</project>
Extending the Model
In our petstore example in the previous section we added attributes to <module> like type=java and also added elements to <module> like <source location="src"/>
If you reference a property off of ${module} in the template, modularity will first look to see if an attribute exists off of <module> so ${module.type} will be a STRING and evaluate to a java.
If an attribute does not exist, modularity will look to see if an element exists off of <module>. In this case modularity will return a LIST of MAPS. The attributes of the element will be available as properties from the MAP. For example take the following template:
<project name="${module.name}.compile" default="compile"
basedir="${module.dir}">
<target name="compile">
<javac destdir="${module.buildDir}/classes">
<#list module.source as source>
<src path="${source.location}"/>
</#list>
</javac>
</target>
</project>
This will generate the following ant build file when the object model for the module 'petstore-core' is passed through:
<project name="petstore-core.compile" default="compile"
basedir="/projects/petstore/petstore-core">
<target name="compile">
<javac destdir="/projects/petstore/modular-build/modules/petstore-core/classes">
<src path="src"/>
</javac>
</target>
</project>
Tasks
There are three modularity tasks that are covered below:
- m:build - The main modularity task. Generates Ant build files from freemarker templates. The freemarker model is built up from the project metadata. The ant build files then get executed in order.
- m:meta2idea - Helper task to generate IntelliJ IDEA project and module files.
- m:version - Helper task to automate the incrementing of a projects version when doing a build.
m:build
Below is an overview of what m:build does. It generates an object model from the GPC, the module group and all the module metadata.xml files. It uses this model to populate various freemarker templated Ant build files. It then executes each resulting Ant build file in order.
|
m:build has the following parameters:
- buildDir - root build directory
- gpcDir - the location of the GPC directory
- gpcFilename - the location of the GPC xml descriptor
- metadata - the location of the root module or module groups metadata.xml
- workspaceDir - the projects root directory (modules are looked for in subdirectories off of here)
Default values are given below each parameters:
- buildDir - modular-build
- gpcDir - ${workspaceDir}/../gpc
- gpcFilename - ${workspaceDir}/gpc.xml
- metadata - metadata.xml (in the current directory)
- workspaceDir - modularity will keep looking up the directory hierarchy until it finds a file called .modularityBaseDir, it will set the workspaceDir to the containing directory.
m:build has one allowed nested element m:execute. m:execute takes the following paramaters:
- module - the module that should be used to populate the freemarker object model. If not included m:execute will loop over every module in the dependency graph and execute each one (starting with the modules at the bottom of the dependency graph)
- templateFilename - the template used to generate the resulting ant build file
m:execute has one allowed nested element parameter. The list of parameters gets passed to the template via the freemarker object model. parameter takes the following parameters:
- name - the parameter name
- value - the parameter value
Two examples of m:build are given below:
The simplest use of m:build
This m:build example will:
1. compile and jar all modules in the dependency graph
<m:build>
<m:execute templateFilename="templates/compile-and-jar.ftl"/>
</m:build>
A more complex use of m:build
This m:build example will in order:
1. retrieve all missing gpc packages (using the template retrieve-gpc.gtl)
2. compile and jar all modules in the dependency graph
3. run junit tests against all the modules in the dependency graph
4. execute the template 'web-release.ftl' for a single module 'petstore-web'
<m:build metadata="metadata.xml" buildDir="modular-build"
gpcDir="../../gpc" gpcFilename="../gpc.xml" workspaceDir="../">
<m:execute module="petstore" templateFilename="templates/retrieve-gpc.ftl">
<parameter name="host" value="gpc.excentric.com">
<parameter name="username" value="gpc">
<parameter name="password" value="secret">
<parameter name="gpcLocation" value="/projects/gpc">
</m:execute>
<m:execute templateFilename="templates/compile-and-jar.ftl"/>
<m:execute templateFilename="templates/run-junit-tests.ftl"/>
<m:execute module="petstore-web" templateFilename="templates/web-release.ftl"/>
</m:build>
m:meta2idea
This task generates IntelliJ IDEA project and module files from the modularity metadata. It uses freemarker templates similar to m:build.
m:meta2idea takes the following parameters:
- metadata - the location of the root module groups metadata.xml
- gpcDir - the location of the GPC directory
- gpcFilename - the location of the GPC xml descriptor
- projectTemplateFile - the location of the .ipr project template
- javaModuleTemplateFile - the location of the java .iml module template
- webModuleTemplateFile - the location of the web .iml module template
- j2meModuleTemplateFile - the location of the j2me .iml module template
An example of m:meta2idea is given below:
<m:meta2idea metadata="metadata.xml"
gpcDir="../../gpc"
gpcFilename="../gpc.xml"
projectTemplateFile="../templates/project.ipr.ftl"
javaModuleTemplateFile="../templates/java-module.iml.ftl"
webModuleTemplateFile="../templates/web-module.iml.ftl"/>
</m:meta2idea>
m:version
This is a helper task to modify the version attribute of a moduleGroup.
m:version takes the following parameters:
- metadata - the location of the module groups metadata.xml
- addDevelopmentLabel - if true will add a '-DEV' to the version
- removeDevelopmentLabel - if true will remove an existing '-DEV' from the version
- incrementVersion - if true will increment the minor most digit of the version
Examples of m:version are given below:
version 1.0.2-DEV to 1.0.2 <m:version removeDevelopmentLabel="true"/>
version 1.0.2 to 1.0.3-DEV <m:version addDevelopmentLabel="true" incrementVersion="true"/>
Core Templates
retrieve-gpc.ftl
compile-and-jar.ftl
run-junit-tests.ftl
create-web-release.ftl
create-app-release.ftl
Extra Templates
run-junit-tests-with-cobertura.ftl
run-cobertura-report.ftl
structure101.ftl
run-exactor-tests.ftl
Contact Information
Sean Coughlan <excentric@gmail.com>