Skip to content

Project 64 - Test AI project generation prompts

Prompt quality

  • Paste the exact custom prompts you used for all 3 projects.
    • Branding OLX: Create a project based on the OLX developer docs. Use very similar branding and structure. The project should support ukrainian and english languages with ukrainian being a default one
    • Branding Mercedes: Create a project based on Mercedes-Benz developer portal. Extract logo from the attached site. Please make sure you follow the exact branding of the developer portal
    • Starter with versions: Generate a minimum project that uses versioned markdown content
  • Which prompt was easiest for the AI to interpret, and why? - Branding prompts.
  • Which prompt was too vague, too broad, or missing important context? - Starter with versions. AI probably has no knowledge about versions.
  • Which prompt details most clearly influenced the generated project? - Reference links, probably.

Output quality

  • Did the generated project feel usable as a first draft? - yes
  • Was the structure clear and appropriate for the intended audience? - ok as a starting point
  • Was the generated content accurate, specific, and free of obvious filler? - seem to reflect the source link that we provided at the first glance - but only a couple of pages per generated project
  • What would you need to fix before sharing the project with someone else? - add more content, fix minor UI issues, fix translations for default language

Feature controls

  • Did Add features feel predictable? - Yes
  • Were feature suggestions relevant to the typed prompt? - Yes
  • Did adding and removing features behave as expected? - No (More info in debried above)
  • Which features did you choose for each project?
    • Localization
    • AI Assistance

Resource controls

  • Did Add resources clearly separate links, OpenAPI files, logos, images, and Markdown files? - Yes
  • Did the generated output visibly reflect your uploaded resources and links? - Yes
  • Did the OpenAPI file produce a useful API reference or API-focused structure? - Yes
  • Did the brand challenge use the chosen brand color palette? - Overall yes but for one project the color of navbar was wrong

Overall

What made you smile?

  • How fast it becomes to create a website in comparison to do that manually

What should we change before wider rollout?

  • Localization functionallity was not translated text on the .tsx page (translations.yaml wasn't created for the default language)

  • "@redocly/theme": "latest" was added to package.json - not needed

  • Add functionallity to edit the generated project after initial setup

  • When the project is initially generated, the added files/links become hidden after clicking elsewhere.

    focus.pngno focus.png
  • Remove unnecessary functionality, such as rules, from the default-generated redocly.yaml file.

  • The feature selection changes randomly while you type the prompt. Even the features you have manually selected or removed can be randomly removed or added back.

    project-ai-creation-random-features.gif
  • Link generated in the tsx file points to the file but it is invalid, the links on the React files should use slugs (<OutlineBtn as={Link} to="/apis/dynamic_wiring_diagram.yaml">View API reference</OutlineBtn> -> /apis/dynamic_wiring_diagram)