In Portkey AI, the essential component replacing the Gateway Framework is Guardrails, which enhances the reliability and safety of interactions with large language models (LLMs). Guardrails help ensure that both requests and responses conform to established standards, thereby minimizing the potential risks linked to unpredictable or harmful LLM outputs.
Portkey AI provides an integrated, fully-guardrailed platform that operates in real-time to guarantee compliance with all required standards during LLM interactions. This is crucial, given the inherent fragility of LLMs, which can fail in unexpected ways. Common failures can occur due to API downtime or unexplainable error codes like 400 or 500. More problematic, however, are scenarios where a response with a 200 status code disrupts an application’s functionality due to incorrect or mismatched output.
The Guardrails in the Gateway Framework are specifically designed to address validation challenges for both input and output against predefined criteria. This system employs a variety of validation methods, including regular expression matching, JSON schema validation, and code detection in languages such as SQL, Python, and TypeScript. In addition to these standard checks, Portkey AI also incorporates LLM-based Guardrails that can identify gibberish or detect prompt injection attempts, thus safeguarding against more subtle failures. Currently, more than 20 types of Guardrail checks are supported, each customizable to meet specific needs.
Portkey AI seamlessly integrates with any Guardrail platform, including Aporia, SydeLabs, and Pillar Security. By inputting API keys, users can incorporate policies from these platforms into their Portkey API calls. Implementing Guardrails is straightforward, involving four key steps: creating checks, defining associated actions, enabling the Guardrails through configurations, and linking these configurations to requests. Users can create a Guardrail by selecting from available checks, followed by specifying actions that should be taken based on the outcomes. Actions may include logging results, rejecting requests, generating evaluation datasets, switching to alternative models, or reattempting requests.
The Portkey Guardrail system is highly configurable, allowing organizations to determine the course of action based on various check outcomes. For example, if a check fails, the configuration can be set up to either halt the request entirely or proceed with a specific status code. This flexibility is crucial for organizations aiming to balance security and operational efficiency.
A powerful feature of Portkey’s Guardrails is their integration with the overall Gateway Framework, which manages request processing. This orchestration allows for either asynchronous or synchronous execution of Guardrails. In asynchronous mode, Portkey logs the outcomes without affecting the request, while in synchronous mode, the Guardrail’s verdict directly influences request handling. For instance, a synchronous check may return a specific status code, like 446, indicating that the request should not be processed if it fails.
Portkey AI maintains detailed logs of Guardrail results, including counts of passed and failed checks, the duration of each check, and feedback for each request. This logging capability is crucial for organizations developing evaluation datasets to enhance AI model quality and bolster protection through Guardrails.
In summary, the Guardrails within Portkey AI’s Gateway Framework represent a robust solution to the inherent risks of deploying LLMs in a production setting. With comprehensive checks and corresponding actions, Portkey ensures that AI applications remain secure, compliant, and resilient against the unpredictable behavior of LLMs.
