generateSite

Jeremie Bresson Jody Winter Ralf D. Müller Schrotti Ralf D. Müller

8 minutes to read

About This Task

When you have only one document, the output of generateHTML might meet your requirements. But as your documentation grows, and you have multiple documents, you will need a microsite which bundles all of the information. The generateSite task uses jBake to create a static site with a landing page, a blog and search.

Pages

The microsite is page-oriented, not document-oriented. If you have already organized your documents by chapter, use them as pages to create a great user experience. The arc42-template sources are a good example.

To include a page in the microsite, add a metadata header to it.

page metadata
:jbake-menu: arc42
:jbake-title: Solution Strategy
:jbake-order: 4
:jbake-type: page_toc
:jbake-status: published
:filename: 015_tasks/03_task_generateSite.adoc

:toc:

[[section-solution-strategy]]
=== Solution Strategy

Here is an overview of each element:

jbake-menu

The top-level menu’s code for this page. Defaults to the top-level folder name (without the order prefix) of the .adoc file within the docDir. Example: if the top-level folder name is 10_news it will default to the value news.

For each code, the display text and the order in the top-level menu can be configured.

jbake-title

The title to be displayed in the drop-down top-level menu. Defaults to the first headline of the file.

jbake-order

Applies a sort order to drop-down entries. Defaults to a prefixed file number, such as 04_filename.adoc or to the prefixed number of the second-level folder name. When nothing is defined the default value is -1 or -987654321 for index pages.

jbake-type

The page type. Controls which template is used to render the page. You will mostly use page for a full-width page or page_toc for a page with a table of contents (toc) rendered on the left. Defaults to page_toc.

jbake-status

Either draft or published. Only published pages will be rendered. Defaults to published for files with a jbake-order and draft for files without jbake-order or files prefixed with _.

filename

Required for edit and feedback links (coming soon). Defaults to the filename :-).

ifndef

Fixes the imagesdir according to the nesting level of your docs folder. Defaults to the main docDir/images.

toc

For :jbake-type: page_toc, you need this line to generate the toc.

Note
Start your pages with a == level headline. You can fix the level offset when you include the page in a larger document with include::chapter.adoc[leveloffset=+1].

Configuration

The configuration follows the convention-over-configuration approach. If you follow the conventions, you don’t have to configure anything. But if you want to, you can override the convention behaviour with a configuration.

Menu

Navigation elements

The navigation is organized with following elements:

  • A top level menu.

  • For each item of this top level menu, a section sidebar on the left.

The location of a page in the top-level menu and in the section sidebar depends from:

  • Its location in the folder structure

  • Page attributes

  • Site configurations

Example:

src/docs/
├── 10_foo
│   ├── 10_first.adoc
│   └── 20_second.adoc
└── 20_bar
    ├── 10_lorem.adoc
    ├── 20_ipsum
    │   ├── 10_topic-A.adoc
    │   └── 20_topic-B.adoc
    └── 30_delis
        ├── 10_one.adoc
        ├── 20_two.adoc
        └── index.adoc

The top level folders (10_foo and 20_bar) are used to determine to which menu-code the page belongs (foo and bar, unless overridden by the :jbake-menu: inside each page).

In the section sidebar, the navigation tree is determined by the folder structure. Folders are nodes in the sidebar tree. Each node can contains pages (leafs) or other folders (child nodes). The order is controlled by the prefix of the file or folder name (unless overridden by the :jbake-order: inside each page).

When an index page is present (like 20_bar/30_delis/index.adoc in the example) then the navigation tree node corresponds to this index page (you can click on it and the title is taken from the page). When this index.adoc does not declare a specific order with :jbake-order: then the order of the parent folder (for the example: 30 because the folder is named 30_delis).

When the index page is absent (like there is no 20_bar/20_ipsum/index.adoc in the example) then the name of the folder is used to create the node, and you can not click on the node because no pages is associated with this node. You can still define the order with the name (for the example 20 because the folder is named 20_ipsum).

When there is no sub-folder, only a flat list of pages is created.

Note
When an index.adoc page is defined inside the top level folder (like: 10_foo/index.adoc or 20_bar/index.adoc) then the page will listed in the section navigation tree in the sidebar as any other regular page. By default it will be the first element of the tree, unless the value is overridden by a :jbake-order: attribute.
Configuring the top level menu

The :jbake-menu: is only the code for the menu entry to be created. You can map these codes to menu entries through the configuration (microsite-Section) in the following way:

menu = [code1: 'Some Title 1', code2: 'Other Title 2', code3: '-']

When no mapping is defined in the configuration file, the code is used as title.

The menu configuration is also impacting the display order.

If you have four files, located in following folder structure:

src/docs
├── code1
│   ├── demo1.adoc
│   └── demo2.adoc
└── code3
    ├── demo3.adoc
    └── _demo4.adoc

Where demo1.adoc and demo3.adoc contain no :jbake-menu: header, demo2.adoc contains :jbake-menu: code2, then:

  • demo1.adoc will have a menu-code of code1 because it is located in the folder code1. This code is translated through the configuration to the menu named Some Title 1.

  • demo2.adoc is in the same folder, but the :jbake-menu: attribute has a higher precedence which results in menu-code code2. This code is translated through the configuration to the menu named Other Title 2.

  • demo3.adoc will have a menu-code code3 because it is located in the folder code3. This code is translated through the configuration to the special menu - which will not be displayed. This is an easy way to hide a menu in the rendered microsite.

  • _demo4.adoc starts with an underscore _ and thus will be handled as draft (:jbake-status: draft). It will not be rendered as part of any menu, but it will be available in the microsite as "hidden" _demo4-draft.html. Feel free to remove these draft renderings before you publish your microsite.

In the column on the right, links are driven by the values defined in docToolchainConfig.groovy.

  • "Improve this doc": displayed when gitRepoUrl is set.

  • "Create an issue": displayed when issueUrl is set.

Configuring the JBake plugin

Behind the scene the generateSite task is relying on Jbake.

In the docToolchainConfig.groovy it is possible to amend the configuration of the jbake gradle plugin:

  • Add additional asciidoctorj plugins (add dependencies to the jbake configuration)

  • Add additional asciidoctor attributes

jbake configuration
//customization of the Jbake gradle plugin used by the generateSite task
jbake.with {
    // possibility to configure additional asciidoctorj plugins used by jbake
    plugins = [ ]

    // possibiltiy to configure additional asciidoctor attributes passed to the jbake task
    asciidoctorAttributes = [ ]
}

The plugins are retrieved from a repository (by default maven-central) configured with project property depsMavenRepository. When a repository requiring credentials is used the properties depsMavenUsername and depsMavenPassword can be set as well.

Templates and Style

The jBake templates and CSS are hidden for convenience.

The basic template uses Twitter Bootstrap 5 as its CSS framework. Use the copyThemes task to copy all hidden jBake resources to your project. You can then remove the resources you don’t need, and change those you want to change.

Note
copyThemes overwrites existing files, but because your code is safely managed using version control, this shouldn’t be a problem.

Landing Page

Place an index.gsp page as your landing page in src/site/templates. This landing page is plain HTML5 styled with Twitter Bootstrap. The page header and footer are added by docToolchain. An example can be found at copyThemes or on GitHub.

Blog

The microsite also contains a simple but powerful blog. Use it to inform your team about changes, as well as architecture decision records (ADRs).

To create a new blog post, create a new file in src/docs/blog/<year>/<post-name>.adoc with the following template:

blog post template
:jbake-title: <title-of your post>
:jbake-date: <date formatted as 2021-02-28>
:jbake-type: post
:jbake-tags: <blog, asciidoc>
:jbake-status: published

:imagesdir: ../../images

== {jbake-title}
{jbake-author}
{jbake-date}

<insert your text here>

The microsite does not have its own local search. But it does have a search input field which can be used to link to another search engine.

CI/CD

When running in an automated build, set the environment variable DTC_HEADLESS to true or 1. This stops docToolchain from asking to install the configured theme, and it will simply assume that you do want to install it. You can also avoid the theme downloading with every build by copying the themes folder from $HOME/.doctoolchain/themes to the corresponding folder in your build container.

Further Reading and Resources

Read about the previewSite task here.

Source

scripts/generateSite.gradle
import groovy.util.*
import static groovy.io.FileType.*

buildscript {
    repositories {
        maven {
            credentials {
                username mavenUsername
                password mavenPassword
            }
            url mavenRepository
        }
    }
    dependencies {
        classpath 'org.asciidoctor:asciidoctorj-diagram:2.0.2'
    }
}
repositories {
    maven {
        credentials {
            username depsMavenUsername
            password depsMavenPassword
        }
        url depsMavenRepository
    }
}
dependencies {
    jbake 'org.asciidoctor:asciidoctorj-diagram:2.0.2'
    jbake 'io.pebbletemplates:pebble:3.1.2'
    config.jbake.plugins.each { plugin ->
        jbake plugin
    }
}

apply plugin: 'org.jbake.site'
apply plugin: 'org.gretty'

def color = { color, text ->
    def colors = [black: 30, red: 31, green: 32, yellow: 33, blue: 34, magenta: 35, cyan: 36, white: 37]
    return new String((char) 27) + "[${colors[color]}m${text}" + new String((char) 27) + "[0m"
}

jbake {
    version = '2.6.7'
    srcDirName = "${targetDir}/microsite/tmp/site"
    destDirName = "${targetDir}/microsite/output"
    configuration['asciidoctor.option.requires'] = "asciidoctor-diagram"
    config.microsite.each { key, value ->
        configuration['site.'+key-'config.microsite.'] = value?:''
        //println 'site.'+key-'config.microsite.' +" = "+ value
    }

    configuration['asciidoctor.attributes'] = [
        "sourceDir=${targetDir}",
        'source-highlighter=prettify@',
        //'imagesDir=../images@',
        "imagesoutDir=${targetDir}/microsite/output/images@",
        "imagesDir=${config.microsite.contextPath}/images@",
        "targetDir=${targetDir}",
        "docDir=${docDir}",
        "projectRootDir=${new File(docDir).canonicalPath}@",
    ]
    if(config.jbake.asciidoctorAttributes) {
        config.jbake.asciidoctorAttributes.each { entry ->
            configuration['asciidoctor.attributes'] << entry
        }
    }

}
bakePreview {
    port = '8046'
}
gretty {
    httpPort = "${config.microsite.previewPort?:8042}" as Integer
    contextPath = "${config.microsite.contextPath}"
    extraResourceBases = ["${targetDir}/microsite/output"]
}


def prepareAndCopyTheme = {
    //copy internal theme
    println "copy internal theme ${new File(projectDir, 'src/site').canonicalPath}"
    copy {
        from('src/site')
        into("${targetDir}/microsite/tmp/site")
    }
    //check if a remote pdfTheme is defined
    def siteTheme = System.getenv('DTC_SITETHEME')?:""
    def themeFolder = new File(projectDir, "../themes/" + siteTheme.md5())
    try {
        if (siteTheme) {
            println "use siteTheme $siteTheme"
            //check if it is already installed
            if (!themeFolder.exists()) {
                if (System.getenv('DTC_HEADLESS')) {
                    ant.yesno = "y"
                } else {
                    println "${color 'green', """\nTheme '$siteTheme' is not installed yet. """}"
                    def input = ant.input(message: """
${color 'green', 'do you want me to download and install it to '}
${color 'green', '   ' + themeFolder.canonicalPath}
${color 'green', 'for you?'}\n""",
                        validargs: 'y,n', addproperty: 'yesno')
                }
                if (ant.yesno == "y") {
                    themeFolder.mkdirs()
                    download.run {
                        src siteTheme
                        dest new File(themeFolder, 'siteTheme.zip')
                        overwrite true
                    }
                    copy {
                        from zipTree(new File(themeFolder, 'siteTheme.zip'))
                        into themeFolder
                    }
                    delete {
                        delete new File(themeFolder, 'siteTheme.zip')
                    }

                } else {
                    println "${color 'green', """\nI will continue without the theme for now... """}"
                    siteTheme = ""
                }

            }
            //copy external theme
            if (siteTheme) {
                copy {
                    from(themeFolder) {}
                    into("${targetDir}/microsite/tmp/")
                }
                //check if the config has to be updated
                // check if config still contains /** microsite **/
                def configFile = new File(docDir, mainConfigFile)
                def configFileText = configFile.text
                if (configFileText.contains("/** start:microsite **/")) {
                    def configFragment = new File(targetDir,'/microsite/tmp/site/configFragment.groovy')
                    if (configFragment.exists()) {
                        println "${color 'green', """
    It seems that this theme is used for the first time in this project.
    Let's configure it!
    If you are unsure, change these settings later in your config file
    $configFile.canonicalPath
    """}"
                        def comment = ""
                        def conf = ""
                        def example = ""
                        def i = 0
                        configFragment.eachLine { line ->
                            if (line.trim()) {
                                if (line.startsWith("//")) {
                                    conf += "    " + line + "\n"
                                    def tmp = line[2..-1].trim()
                                    comment += color('green', tmp) + "\n"
                                    if (tmp.toLowerCase().startsWith("example")) {
                                        example = tmp.replaceAll("[^ ]* ", "")
                                    }
                                } else {
                                    //only prompt if there is something to prompt
                                    if (line.contains("##")) {
                                        def property = line.replaceAll("[ =].*", "")
                                        if (!example) {
                                            example = config.microsite[property]
                                        }
                                        comment = color('blue', "$property") + "\n" + comment
                                        if (example) {
                                            ant.input(message: comment,
                                                addproperty: 'res' + i, defaultvalue: example)
                                        } else {
                                            ant.input(message: comment,
                                                addproperty: 'res' + i)
                                        }
                                        (comment, example) = ["", ""]
                                        line = line.replaceAll("##.+##", ant['res' + i])
                                        conf += "    " + line + "\n"
                                        i++
                                    } else {
                                        conf += "    " + line + "\n"
                                    }
                                }
                            } else {
                                conf += "\n"
                            }
                        }
                        configFile.write(configFileText.replaceAll("(?sm)/[*][*] start:microsite [*][*]/.*/[*][*] end:microsite [*][*]/", "%%marker%%").replace("%%marker%%", conf))
                        println color('green', "config written\ntry\n ./dtcw generateSite previewSite\nto see your microsite!")
                    }
                    //copy the dummy docs (blog, landing page) to the project repository
                    copy {
                        from(new File(themeFolder, 'site/doc')) {}
                        into(new File(docDir, inputPath))
                    }
                }
            }
        }
    } catch (Exception e) {
        println color('red', e.message)
        if (e.message.startsWith("Not Found")) {
            themeFolder.deleteDir()
            throw new GradleException("Couldn't find theme. Did you specify the right URL?\n"+e.message)
        } else {
            throw new GradleException(e.message)
        }
    }
    //copy project theme
    if (config.microsite.siteFolder) {
        def projectTheme = new File(new File(docDir, inputPath), config.microsite.siteFolder)
        println "copy project theme ${projectTheme.canonicalPath}"
        copy {
            from(projectTheme) {}
            into("${targetDir}/microsite/tmp/site")
        }
    }

}

def convertAdditionalFormats = {
    if (config.microsite.additionalConverters) {
        File sourceFolder = new File(targetDir, '/microsite/tmp/site/doc')
        sourceFolder.traverse(type: FILES) { file ->
            def extension = '.' + file.name.split("[.]")[-1]
            if (config.microsite.additionalConverters[extension]) {
                def command = config.microsite.additionalConverters[extension].command
                def type = config.microsite.additionalConverters[extension].type
                def binding = new Binding([
                    file  : file,
                    config: config
                ])
                def shell = new GroovyShell(getClass().getClassLoader(), binding)
                switch (type) {
                    case 'groovy':
                        shell.evaluate(command)
                        break
                    case 'groovyFile':
                        shell.evaluate(new File(docDir, command).text)
                        break
                    case 'bash':
                        if (command=='dtcw:rstToHtml.py') {
                            // this is an internal script
                            command = projectDir.canonicalPath+'/scripts/rstToHtml.py'
                        }
                        command = ['bash', '-c', command + ' ' + file]
                        def process = command.execute([], new File(docDir))
                        process.waitFor()
                        if (process.exitValue()) {
                            def error = process.err.text
                            throw new Exception("""
can't convert '${file.canonicalPath-docDir-'/build/microsite/tmp/site/doc'}':
${error}
""")
                        }
                }
            }
        }
    }

}

def parseAsciiDocAttribs = { beforeToc, origText, jbake ->
    def parseAttribs = true
    def text = ""
    origText.eachLine { line ->
        if (parseAttribs && line.startsWith(":jbake")) {
            line = (line - ":jbake-").split(": +", 2)
            jbake[line[0]] = line[1]
        } else {
            if (line.startsWith("[")) {
                // stop parsing jBake-attribs when a [source] - block starts which might contain those attribs as example
                parseAttribs = false
            }
            text += line+"\n"
            //there are some attributes which have to be set before the toc
            if (line.startsWith(":toc") ) {
                beforeToc += line+"\n"
            }
        }
    }
    return text
}

def parseOtherAttribs = { origText, jbake ->
    if (origText.contains('~~~~~~')) {
        def parseAttribs = true
        def text = ""
        origText.eachLine { line ->
            if (parseAttribs && line.contains("=")) {
                line = (line - "jbake-").split("=", 2)
                jbake[line[0]] = line[1]
            } else {
                if (line.startsWith("~~~~~~")) {
                    // stop parsing jBake-attribs when delimiter shows up
                    parseAttribs = false
                } else {
                    text += line + "\n"
                }
            }
        }
        return text
    } else {
        return origText
    }
}

def renderHeader = { fileName, jbake ->
    def header = ''
    if (fileName.toLowerCase() ==~ '^.*(html|md)$') {
        jbake.each { key, value ->
            if (key == 'order') {
                header += "jbake-${key}=${(value ?: '1') as Integer}\n"
            } else {
                if (key in ['type', 'status']) {
                    header += "${key}=${value}\n"
                } else {
                    header += "jbake-${key}=${value}\n"
                }
            }
        }
        header += "~~~~~~\n\n"

    } else {
        jbake.each { key, value ->
            if (key == 'order') {
                header += ":jbake-${key}: ${(value ?: '1') as Integer}\n"
            } else {
                header += ":jbake-${key}: ${value}\n"
            }
        }
    }
    return header
}
def fixMetaDataHeader = {
    //fix MetaData-Header
    File sourceFolder = new File(targetDir, '/microsite/tmp/site/doc')
    logger.info("sourceFolder: " + sourceFolder.canonicalPath)
    sourceFolder.traverse(type: FILES) { file ->
        if (file.name.toLowerCase() ==~ '^.*(ad|adoc|asciidoc|html|md)$') {
            if (file.name.startsWith("_") || file.name.startsWith(".")) {
                //ignore
            } else {
                def origText = file.text
                //parse jbake attributes
                def text = ""
                def jbake = [
                    status: "published",
                    order: -1,
                    type: 'page_toc'
                ]
                if (file.name.toLowerCase() ==~ '^.*(md|html)$') {
                    // we don't have a toc for md or html
                    jbake.type = 'page'
                }
                def beforeToc = ""
                if (file.name.toLowerCase() ==~ '^.*(ad|adoc|asciidoc)$') {
                    text = parseAsciiDocAttribs(beforeToc, origText, jbake)
                } else {
                    text = parseOtherAttribs(origText, jbake)
                }
                def name = file.canonicalPath - (sourceFolder.canonicalPath+File.separator)
                if (File.separator=='\\') {
                    name = name.split("\\\\")
                } else {
                    name = name.split("/")
                }
                if (name.size()>1) {
                    if (!jbake.menu) {
                        jbake.menu = name[0]
                        if (jbake.menu ==~ /[0-9]+[-_].*/) {
                            jbake.menu = jbake.menu.split("[-_]", 2)[1]
                        }
                    }
                    def docname = name[-1]
                    if (docname ==~ /[0-9]+[-_].*/) {
                        jbake.order = docname.split("[-_]",2)[0]
                        docname     = docname.split("[-_]",2)[1]
                    }
                    if (name.size() > 2) {
                        if ((jbake.order as Integer)==0) {
                            // let's take the order from the second level dir or file and not the file
                            def secondLevel = name[1]
                            if (secondLevel ==~ /[0-9]+[-_].*/) {
                                jbake.order = secondLevel.split("[-_]",2)[0]
                            }
                        } else {
                            if (((jbake.order?:'1') as Integer) > 0) {
                                //
                            } else {
                                jbake.status = "draft"
                            }
                        }
                    }
                    if (jbake.order==-1 && docname.startsWith('index')) {
                        jbake.order = -987654321 // special 'magic value' given to index pages.
                        jbake.status = "published"
                    }
                    // news blog
                    if (jbake.order==-1 && jbake.type=='post') {
                        jbake.order = 0
                        try {
                            jbake.order = Date.parse("yyyy-MM-dd", jbake.date).time / 100000
                        } catch ( Exception e) {
                            System.out.println "unparsable date ${jbake.date} in $name"
                        }
                        jbake.status = "published"
                    }
                    def leveloffset = 0
                    if (file.name.toLowerCase() ==~ '^.*(ad|adoc|asciidoc)$') {
                        text.eachLine { line ->
                            if (!jbake.title && line ==~ "^=+ .*") {
                                jbake.title = (line =~ "^=+ (.*)")[0][1]
                                def level = (line =~ "^(=+) .*")[0][1]
                                if (level == "=") {
                                    leveloffset = 1
                                }
                            }
                        }
                    } else {
                        if (file.name.toLowerCase() ==~ '^.*(html)$') {
                            if (!jbake.title) {
                                text.eachLine { line ->
                                    if (!jbake.title && line ==~ "^<h[1-9]>.*</h.*") {
                                        jbake.title = (line =~ "^<h[1-9]>(.*)</h.*")[0][1]
                                    }
                                }
                            }
                        } else {
                            // md
                            if (!jbake.title) {
                                text.eachLine { line ->
                                    if (!jbake.title && line ==~ "^#+ .*") {
                                        jbake.title = (line =~ "^#+ (.*)")[0][1]
                                    }
                                }
                            }
                        }
                    }
                    if (!jbake.title) {
                        jbake.title = docname
                    }
                    if (leveloffset==1) {
                        //leveloffset needed
                        // we always start with "==" not with "="
                        // only used for adoc
                        text = text.replaceAll("(?ms)^(=+) ", '$1= ')
                    }
                    if (config.microsite.customConvention) {
                        def binding = new Binding([
                            file  : file,
                            sourceFolder : sourceFolder,
                            config: config,
                            headers : jbake
                        ])
                        def shell = new GroovyShell(getClass().getClassLoader(), binding)
                        shell.evaluate(config.microsite.customConvention)
                        System.out.println jbake

                    }
                    def header = renderHeader(file.name, jbake)
                    if (file.name.toLowerCase() ==~ '^.*(ad|adoc|asciidoc)$') {
                        file.write(header + "\nifndef::dtc-magic-toc[]\n:dtc-magic-toc:\n$beforeToc\n\n:toc: left\n\n++++\n<!-- endtoc -->\n++++\nendif::[]\n" + text, "utf-8")
                    } else {
                        file.write(header + "\n" + text, "utf-8")
                    }
                }
            }
        }
    }

}

task generateSite(
    group: 'docToolchain',
    description: 'generate a microsite using jBake.') {
    doLast {
        new File("${targetDir}/microsite/tmp").mkdirs()
        println new File("${targetDir}/microsite/tmp/").canonicalPath

        prepareAndCopyTheme()

        //copy docs
        copy {
            from(new File(docDir, inputPath)) {}
            into("${targetDir}/microsite/tmp/site/doc")
        }

        // if configured, convert restructuredText or anything else
        convertAdditionalFormats()

        // convention over configuration
        fixMetaDataHeader()

    }
}
task previewSite(
    group: 'docToolchain',
    dependsOn: [],
    description: 'start a little webserver to preview your Microsite',
) {
    if (new File("${targetDir}/microsite/output").exists()) {
        finalizedBy 'jettyRun'
    }
    doLast {
        if (new File("${targetDir}/microsite/output").exists()) {
            // everything is fine
        } else {
            throw new GradleException(""">
> Microsite not built yet, please run './dtcw generateSite' first
>""")
        }
    }
}
task copyImages(type: Copy) {
    config.imageDirs.each { imageDir ->
        from(new File (new File(docDir, inputPath),imageDir)) {}
        logger.info ('imageDir: '+imageDir)
        into("${targetDir}/microsite/output/images")
    }
    config.resourceDirs.each { resource ->
        from(new File(file(srcDir),resource.source))
        logger.info ('resource: '+resource.source)
        into("${targetDir}/microsite/output/" + resource.target)
    }
}

bake.dependsOn copyImages
generateSite.finalizedBy bake