generateContent

Johannes Thorn PacoVK

3 minutes to read

About This Task

This task makes use of openAI to render documentation into your input Asciidoc files. It can help you to get started with the outline of your documentation or even with real content. This task implements OpenAI completions. Simply put an asciidoc comment block that starts with AI: followed by your description what you need.

Note
 You need a valid API-Key from OpenAI to execute this task. Be aware that whatever content you use to describe what you want, OpenAI will process this data!

Configuration

Config.groovy
// Configuration for openAI related tasks
openAI = [:]

openAI.with {
    // This task requires a person access token for openAI.
    // Ensure to pass this token as parameters when calling the task
    // using -PopenAI.token=xx-xxxxxxxxxxxxxx

    //model = "text-davinci-003"
    //maxToken = '500'
    //temperature = '0.3'
}
Warning
 Token is a credential, it is best not to put it into your config, but pass this parameter to the task using -PopenAI.token=xx-xxxxxxxxxxxxx at task execution.

Example

Given a config in your project, that specifies the following inputFiles:

docToolchainConfig.groovy
/** config redacted **/
inputFiles = [
	[file: 'example/example.adoc', formats: ['html','pdf', 'docbook']],
        /** inputFiles **/
]
/** config redacted  **/

You could generate a whole file, e.g. an outline for your documentation.

example/example.adoc
////
AI: Create an outline for a documentation of a Java based microservice called fraud-detector. The documentation should follow the Divio structure. Use Asciidoc syntax for the output. The result should define a toc.
////
Note
 The result varies based on your input you give, the more specific you are, the better suited the output.

The result could look like the following:

example/example.adoc
= Fraud-Detector
:toc:

== Overview
* Description of fraud-detector microservice
* Abstract view of architecture
* List of core libraries used

== Getting Started
* Setup instructions
* How to add it as a dependency
* Configuration instructions

== Architecture
* In-depth view of architecture
  * Components
  * Services
  * Data Structure

== Development
* Contribution Guidelines
* Coding Standards
* Debugging Suggestions

== Troubleshooting
* Known Issues and Solutions
* Support

== FAQs
* Common Questions and Answers

You can also let just parts of your docs, like installation of basic tools, e.g. JDK 17, generate.

Note
 You can specify as many AI: blocks as you want. They will al be replaced by openAI.
example/example.adoc
== Getting Started

* Setup instructions

////
AI: Write a short how-to install JDK 17 on Windows, Mac and Linux using sdkman. Use Asciidoc syntax for the output.
////

* How to add it as a dependency

The example above, will lead to a result similar to:

example/example.adoc
= Installing JDK 17 using SDKMAN

JDK 17 is a version of the Java Development Kit that is suitable for general use. It can be installed quickly and easily with the help of SDKMAN. This guide includes instructions for installation on Windows, Mac and Linux operating systems.

== Windows

JDK 17 can be installed on Windows with the following commands:

[source,shell]
----
sdk install java 17.0.1-open
----

== Mac

JDK 17 can be installed on Mac with the following commands:

[source,shell]
----
sdk install java 17.0.1-open
----

== Linux

JDK 17 can be installed on Linux with the following commands:

[source,shell]
----
sdk install java 17.0.1-open
----

Source

Show source code of scripts/generateContent.gradle or go directly to GitHub · docToolchain/scripts/generateContent.gradle.
scripts/generateContent.gradle
task generateContent (
    description: 'Generate documentation with the help of OpenAI',
    group: 'docToolchain'
) {
    doLast{
        def token  =  findProperty("openAI.token")?:null
        if(token == null){
            throw new Exception ("""
            >> No Token provided for openAI.
            >> Please specify the token passing the parameter -PopenAI.token=xx-xxxxxxxxxxxxxxxx
            """)
        }
        def model = (findProperty("openAI.model")?:config.openAI.model)?:"text-davinci-003"
        def maxToken = (findProperty("openAI.maxToken")?:config.openAI.maxToken)?:500
        def temperature = (findProperty("openAI.temperature")?:config.openAI.temperature)?:0.3
        logger.info "docToolchain> Generating AI content"
        logger.info "docToolchain> using model ${model}"
        logger.info "docToolchain> limiting output to ${maxToken} tokens"

        def pattern = /\/\/\/\/[\r\n\s]AI:*(.*?)[\r\n\s]*\/\/\/\// // match asciidoc block comments
        def service = new OpenAiService(token, Duration.ofSeconds(60));
        def sourceFiles = config.inputFiles
        srcDir  = "${docDir}/${inputPath}"
        logger.info "docToolchain> Checking for AI comment block in ${sourceFiles}"
        sourceFiles.every {
            def sourceFile = new File(srcDir, it.file)
            def alltext = sourceFile.text

            def matcher = alltext =~ pattern
            matcher.each { match ->
                println("---------")
                println("Request: \n${match[1]}")
                println("---------")
                def completionRequest = CompletionRequest.builder()
                    .prompt(match[1])
                    .model(model)
                    .maxTokens(maxToken)
                    .temperature(temperature)
                    .echo(false)
                    .build();
                def result = service.createCompletion(completionRequest).getChoices()[0]
                def generatedContent = result.text.replaceAll(/^[\s\n\r]+/, '')
                if(result.finish_reason == "length"){
                    generatedContent += "\n// [maxToken ${maxToken} exceeded. Raise to get the full content...]"
                }
                println("Result:")
                println(generatedContent)
                alltext = alltext.replace(match[0], generatedContent)
            }
            sourceFile.write(alltext)
        }
        service.shutdownExecutor();
    }
}