Building a GenAI-assisted automatic story illustrator

Introduction

An illustrated version of "The Gift of the Magi"

Illustrate your own stories for FREE with just a few lines of python code

import requests
import os

URL = "http://195.242.25.198:8020/illustrate/story"
WANDB_API_KEY = "<YOUR-WANDB-API-KEY>"


illustrator_payload = dict(
    story="Once upon a time, in a land far, far away, there was a little girl named Alice. Alice loved to explore the world around her. One day, she decided to go on an adventure to find the most beautiful flower in the world. Alice traveled through forests, over mountains, and across rivers. She met many interesting creatures along the way, like a friendly dragon and a wise old owl. After a long journey, Alice finally found the most beautiful flower in the world. She picked it and brought it back to her home. From that day on, Alice knew that with courage and determination, she could achieve anything.",
    story_title="Alice's Flower Adventure",
    story_author="A.A. Milne",
    story_setting="A magical land filled with talking animals and enchanted creatures",
)

response = requests.post(URL, headers={"wandb-api-key": WANDB_API_KEY}, json=illustrator_payload)
print(response.json())



Acknowledgement

Table of contents

Architecture of the story illustrator

Results

Shortcomings and potential improvements

Currently the character profiles are created on paragarph/story-segment basis. This is not an optimal approach, as it wastes a lot of redundant LLM calls for characters that are present at multiple points in the story across several segments. An easy way to solve this problem is to search the story once for all the characters and store their profiles in the memory. This not only reduces the expenditure in terms of LLM calls, but also makes the character profiles coherent across the entire story. Note that the dress and looks of a character might change in the story, which means we will still need to look for these changes with respect to the characters in each paragraph/story-segment.
For generating both character profiles and the summary we're currently passing the entire story as context to the LLM call. This approach is suboptimal and makes the entire workflow slow and expensive. This could be optimized using a retrieval augmented generation (RAG) workflow where only those segments from the story that are relevant to the current character profile or summary is retrieved and used as context. An advanced retreival technique such as Recursive Abstractive Processing for Tree Organized Retrieval (RAPTOR) could potentially be used to capture both high level and detailed aspects of text, which is particularly useful for complex thematic queries used in this workflow. This would also enable us to reliably scale the auto-illustration architecture for larger stories and novels that would not fit into the context length of the LLM.

The current architecture often misses to include important clues from the paragraph/story-segment such as the story being set during Christmas (in spite of it being explicitly mentioned in the text). This could be solved by prompting explicitly to look for specific visual details, such as what season it is now, or what time of the year it is, or even what time of the day is being mentioned here to ensure details like these are not overlooked in the summary.
Assigning roles or a persona to the system prompt of an LLM is an important technique that can be used to increase the accuracy of its response by asking to respond in a certain style. For example, for the summarization step in the InContextTextToImagePromptGenerator, we ask LLM to play the role of a "helpful assistant to a visionary film director". Similar role assignment could be used in other steps, such as the entity recognition or the character profiling steps to make responses more accurate.
Generating consistent faces corresponding to the characters across all the illustrations is another challenge that needs to be tackled. The cheapest way to achieve this is to "cast" the characters present in the story to an existing famous celebrity/actor whose likeness is known to the text-to-image generation model. This would require some clever post-processing of the summary generated by the InContextTextToImagePromptGenerator such that all references to a particular character with that of the cast. Another way to generate consistent faces more reliably is to use DreamBooth to personalize your text-to-image generation model using 3-5 images of a subject (check this script for DreamBooth with FLUX). If you're using a text-to-image generation service like MidJourney, you can use the character reference feature.

Conclusion