When ChatGPT and other large language models began entering the mainstream two years ago, it quickly became apparent the technology could excel at certain business functions, yet it was less clear how well artificial intelligence could handle more creative tasks.
Sure, generative AI can summarize the content of an article, identify patterns in data, and produce derivative work—say, a song in the style of Taylor Swift or a poem in the mood of Langston Hughes—but can the technology develop truly innovative ideas?
Specifically, Harvard Business School Assistant Professor Jacqueline Ng Lane was determined to find out “how AI handled open-ended problems that haven’t been solved yet—the kind where you need diverse expertise and perspectives to make progress.”
In a working paper published in the journal Organization Science, Lane and colleagues compare ChatGPT’s creative potential to crowdsourced innovations produced by people. Ultimately, the researchers found that both humans and AI have their strengths—people contribute more novel suggestions while AI creates more practical solutions—yet some of the most promising ideas are the ones people and machines develop together.
Lane cowrote the paper with Léonard Bouissioux, assistant professor at the University of Washington’s Foster School of Business; Miaomiao Zhang, an HBS doctoral student, Karim Lakhani, the Dorothy & Michael Hintze Professor of Business Administration at HBS; and Vladimir Jacimovic, CEO and founder of ContinuumLab.ai and executive fellow at HBS.
Crowdsourcing people for ‘moonshots’
Any innovation process usually starts with brainstorming, says Lane, whose research has long looked at how creative ideas are produced.
“You start with defining the problem, then you generate ideas, then you evaluate them and choose which ones to implement.”
“It’s like a funnel,” she says. “You start with defining the problem, then you generate ideas, then you evaluate them and choose which ones to implement.”
Research has shown that crowdsourcing can be an effective way to generate initial ideas. However, the approach can be time-consuming and expensive. Creative teams typically offer incentives to respondents for their ideas. Then teams often must wait for input and then comb through ideas to come up with the most promising leads.
An off-the-shelf large language model such as ChatGPT, however, is free or low cost for end users, and can generate an infinite number of ideas quickly, Lane says. But are the ideas any good?
To find out, Lane and her fellow researchers asked people to come up with business ideas for the sustainable circular economy, in which products are reused or recycled to make new products. They disseminated a request on an online platform, offering $10 for participating and $1,000 for the best idea. Here’s part of their request:
We would like you to submit your circular economy idea, which can be a unique new idea or an existent idea that is used in the industry.
Here is an example: Car sharing in order to reduce the carbon footprint associated with driving. …
Submit your real-life use cases on how companies can implement the circular economy in their businesses. New ideas are also welcome, even if they are “moonshots.”
Seeking creative ideas from ChatGPT
The researchers asked for ideas that would involve “sharing, leasing, reusing, repairing, refurbishing [or] recycling existing materials and products as long as possible.” Suggestions would be scored for uniqueness, environmental benefits, profit potential, and feasibility.
Some 125 people replied with contributions, offering insights from a variety of industries and professional backgrounds. One, for example, proposed a dynamic pricing algorithm for supermarkets to cut down on food waste, while another suggested a mobile app that could store receipts to reduce paper waste.
At the same time, the research team employed prompt engineering techniques to craft a variety of AI prompts. Using these carefully designed prompts, they generated several hundred additional solutions through ChatGPT. The team strategically modified their prompts to:
- Challenge the model to create more ideas.
- Mimic the perspective of someone from a particular industry, job title, and place—a persona.
- Remind the model to provide ideas that reflect the scoring criteria.
The team then recruited some 300 evaluators well-versed in the circular economy to evaluate a randomized selection of the ideas based on the scoring criteria.
People are creative, but AI ideas are more feasible
The evaluators judged the human solutions as more novel, employing more unique “out of the box” thinking. However, they found the AI-generated ideas to be more valuable and feasible.
For example, one participant from Africa proposed creating interlocking bricks using foundry dust and waste plastic, creating a new construction material and cutting down on air pollution at the same time. “The evaluators said, ‘Wow, this is really innovative, but it would never work,’” Lane says.
“We were surprised at how powerful these technologies were.”
One ChatGPT response, meanwhile, created an idea to convert food waste into biogas, a renewable energy source that could be used for electricity and fertilizer. Not the most novel idea, the researchers noted, but one that could be implemented and might show a clear financial return.
“We were surprised at how powerful these technologies were,” Lane says, “especially in these early stages in the creative process.”
How to reach the best solutions
The “best” ideas, Lane says, may come from those in which humans and AI collaborate, with people engineering prompts and continually working with AI to develop more original ideas.
“We consistently achieved higher quality results when AI would come up with an idea and then we had an instruction that said: Make sure before you create your next idea, it’s different from all the ones before it,” Lane explains.
Additional prompts increased the novelty of the ideas, generating everything from waste-eating African flies to beverage containers tracked by smart chips that instantly pay consumers for recycling them.
Based on the findings, the researchers suggest business leaders keep a few points in mind when implementing AI to develop creative solutions:
- Knowing how to ask the right questions is important. Organizations might want to invest in cultivating an “AI-literate” workforce that can understand the capabilities and limitations of AI to generate the most successful ideas.
- Organizations should resist the temptation to rely excessively on AI. That could “dumb down” the overall level of creative output over time, leading to more incremental improvements than radical breakthroughs, the team says.
- People should view generative AI models as collaborative tools. In a sequential approach, humans could brainstorm solutions, then submit them to AI to refine them and increase their value and feasibility. Alternatively, humans could work more iteratively with AI, constantly shaping and improving the ideas it provides.
The most productive way to use generative AI, the research suggests, is to combine the novelty that people excel at with the practicality of the machine. Says Lane, “We still need to put our minds toward being forward-looking and envisioning new things as we are guiding the outputs of AI to create the best solutions.”
You Might Also Like:
Feedback or ideas to share? Email the Working Knowledge team at hbswk@hbs.edu.
Image: Image by HBSWK with assets from AdobeStock/phaisarnwong2517/Ирина Гутыряк/Ruslan Gilmanshin
