The Prompt Engineering Process

I would like to propose that prompt engineering is distinct from prompt crafting. Let’s discuss how and why the two differ.

I consider prompt crafting to be one of those spontaneous, on-the-fly creations consisting of a direct instruction or query to an LLM for immediate results. An example of prompt crafting is: “Please give me an executive summary of the following text”. This prompt would likely yield a usable summary of some indeterminate length that may or may not include bullet-points. In prompt crafting, you expect useful results, however, they may not be reliable or repeatable or exactly what you want.

Prompt engineering, on the other hand, is a process that effectively “programs” LLMs to produce content that aims to be both repeatable and reliable while fulfilling a specific function. As a process, it is both iterative and results-focused. Let’s look closer at what this process entails.

An Iterative Process

Firstly, we must understand that prompt engineering is not a linear process with a defined start and finish, but rather a continuous cycle of improvement. The goal is to refine the instructions given to an LLM to achieve a desired outcome. This refinement requires continuous adjustments based on the LLM's output, feedback from experts, and analysis of performance metrics.

Secondly, we can identify six main steps in the process:

• 1. Problem Definition: The first step is to clearly define the goals you want to achieve with the LLM and the specific task it should perform. This also involves identifying potential challenges related to the task or domain.
• 2. Domain Understanding and Data Analysis: After defining the problem, we combine input from domain experts and gather relevant data to inform prompt design. This step involves analyzing existing documentation, project specifications, and research papers to identify patterns, keywords, and other relevant information.
• 3. Prompt Design and Technique Selection: At this point, we draft the initial prompt based on the defined objectives, domain knowledge, and data analysis. The prompt should be clear, specific, contextually relevant, and targeted to address the identified challenges. The choice of prompting techniques and/or patterns employed will depend on the complexity of the task.
• 4. Evaluation and Refinement: Once the initial prompt is designed, we test it with the LLM, and evaluate the output. This includes checking for accuracy, relevance, conciseness, potential biases, and alignment with the defined goals. Based on this evaluation, we refine the prompt and improve it iteratively. This might involve adjusting the wording, adding more context, or switching to a different prompting technique.
• 5. Expert Feedback and Collaboration: Throughout the process, we seek feedback from subject matter experts and collaborate with them to improve the prompt and ensure the LLM's output meets the desired standards.
• 6. Documentation: As we refine the prompt and see improvements in the LLM output, we document what we have learned, what succeeded, what didn't. This documentation serves as a valuable resource for future prompt engineering endeavors and helps in sharing knowledge within the field.

What the Process Looks Like

This diagram is a simplified representation as the process typically involves a dynamic interplay between different stages.

For instance, evaluating the LLM output might highlight the need to revisit the problem definition or the data analysis stage. Similarly, expert feedback could prompt changes in both prompt design and technique selection.

This continuous feedback loop and the iterative nature of refinement are crucial aspects of prompt engineering, enabling the development of increasingly effective prompts that harness the full potential of LLMs.

Prompt crafting—everyone is doing this. Prompt engineering—this takes expertise.