Meet JASON, the AI that can explore concepts in JSON structure!

[mrmemo]

JASON, the GPT-4 AI that can explore concepts in JSON structure

Problem Summary:

Out of the box, GPT4 doesn't do "JSON" structures in great detail. It will generally favor breadth over depth, and frequently duplicate sub-structures within that object. It will also try to give example values for the object, in lieu of exploring greater detail of structure. This means that if you ask GPT-4 to generate a hierarchy in key-value pairs like JSON, it'll generally "skim over" the top layer of complexity, rather than "diving into" the object. It will also waste tokens generating example values for keys, and duplicate structures within its response. This can limit its ability to explore complex concepts as hierarchies.

Here's an example:

PROMPT:

Turn the U.S. Government into JSON format.

BASE GPT-4 RESPONSE:

{
  "us_government": {
    "branches": [
      {
        "name": "Legislative",
        "description": "Responsible for creating and passing laws",
        "bodies": [
          {
            "name": "Senate",
            "total_members": 100
          },
          {
            "name": "House of Representatives",
            "total_members": 435
          }
        ]
      },
      {
        "name": "Executive",
        "description": "Responsible for enforcing laws and running the government",
        "leader": {
          "title": "President",
          "name": "John Doe"
        },
        "departments": [
          {
            "name": "Department of State",
            "abbreviation": "DoS",
            "head": "Secretary of State"
          },
          {
            "name": "Department of the Treasury",
            "abbreviation": "DoT",
            "head": "Secretary of the Treasury"
          },
          {
            "name": "Department of Defense",
            "abbreviation": "DoD",
            "head": "Secretary of Defense"
          },
          {
            "name": "Department of Justice",
            "abbreviation": "DoJ",
            "head": "Attorney General"
          },
          {
            "name": "Department of Homeland Security",
            "abbreviation": "DHS",
            "head": "Secretary of Homeland Security"
          }
        ]
      },
      {
        "name": "Judicial",
        "description": "Responsible for interpreting laws and administering justice",
        "court": {
          "name": "Supreme Court",
          "total_justices": 9,
          "chief_justice": "John Roberts"
        }
      }
    ]
  }
}

We can see that the model natively returns a very cursory review of the subject matter, and sometimes includes what it thinks are correct values for the keys. But this behavior limits its ability to consider complexity and depth.

"JASON'S" Functionality:

JASON iteratively considers the complexity at each level of detail in the JSON structure. This iterative process ensures that JASON will return consistent and predictable results, as well as maximizes the detail which JASON uses to consider sub-structures.

In contrast to base-GPT-4, JASON will explore significantly greater detail of conceptual complexity. This would enable much more robust hypothesis generation within these potentially abstract structures.

JASON'S RESPONSE (V1):

{
  "US Government": {
    "Branches": {
      "Legislative": {
        "Congress": {
          "Senate": {
            "Members": 100,
            "Majority Party": "",
            "Minority Party": ""
            },
          "House of Representatives": {
            "Members": 435,
            "Majority Party": "",
            "Minority Party": ""
            }
          }
        },
        "Executive": {
          "President": {
            "Name": "",
            "Party": "",
            "Term Start": "",
            "Term End": ""
           },
          "Vice President": {
            "Name": "",
            "Party": "",
            "Term Start": "",
            "Term End": ""
          },
          "Cabinet": {
            "Members": [""],
            "Departments": [""]
          }
        },
        "Judicial": {
          "Supreme Court": {
            "Justices": 9,
            "Chief Justice": "",
            "Associate Justices": [""]
          },
          "Lower Courts": {
            "Federal Courts": {
              "District Courts": [""],
              "Circuit Courts": [""]
            }
          }
        }
      },
      "Elections": {
        "Presidential Election": {
          "Date": "",
          "Candidates": [""],
          "Results": {}
        },
        "Midterm Election": {
          "Date": "",
          "Candidates": [""],
          "Results": {}
        },
        "State Elections": {
          "Governor Elections": {},
          "State Legislature Elections": {},
          "Local Elections": {}
        }
      },
    "Budget": {
      "Total Budget": 0,
      "Revenue": {
        "Taxes": 0,
        "Borrowing": 0,
        "Other": 0
      },
      "Expenditures": {
        "Taxes": 0,
        "Borrowing": 0,
        "Other": 0
      }
    }
  }
}

In my example, JASON does a few things better than base-GPT-4:

• Pursues much more detailed and relevant elements within each key
• Correctly condenses frequently-used substructures -- the "Elections" substructures within "State Elections", e.g.
• Considers additions at every iteration of the process, further expanding detail

Where do we go next?

I'm currently working to compress the intermediary output that JASON generates while iterating. This output is necessary to ensure that it follows the process correctly, as well as providing it a way to refer back and structure the final output in more proper JSON format. This intermediary process, while robust and exhaustive, burns a fair number of tokens.

However, it's possible that some (or even all) of this could be offloaded to GPT-3.5, with enough prompt engineering. For now, I'm confident that I can get this functionality with few-shot priming in GPT-4. At this stage, I believe it could readily be incorporated into the AutoGPT codebase as an "AI Companion Method".

Deployment

Deployment would be as simple as starting a new Session with the desired model, priming it with the necessary initial prompts, and querying it conversationally.

Code Exposure

I don't want to push this without a demand, so I'm interested to know: is this functionality worth expanding?

[renxida]

I like this. Very very great for downstream processing. There's a way to specify a schema right?