Adding documents with grounding module is broken since 1.18.0

[mweiler-sap]

Describe the Problem/Bug

Since the release of ai-sdk-java 1.18.0, adding new documents to an existing collection does not work anymore with the grounding module. The request to grounding/create fails by sending a response HTTP/1.1 500 Internal Server Error which the AI SDK then fails to process, resulting in an IO Exception: java.io.IOException: Failed to process response: HTTP/1.1 500 Internal Server Error
The same error still occurs with the latest version (1.20.0)

Steps to Reproduce

• Make sure the resource group "sample-resource-group" exists already in AI Core

• Make sure the collection with a known uuid in the variable collectionId already exists in AI Core

• Run the following snippet:

  List<BaseDocument> baseDocuments = List.of(BaseDocument.create()
                        .chunks(List.of(TextOnlyBaseChunkCreate.create().content("sample content")
                                .metadata(List.of(VectorKeyValueListPair.create().key("key").value("value")))))
                        .metadata(List.of(VectorDocumentKeyValueListPair.create().key("key").value("value"))));
  
                // Create documents by calling vector api
                DocumentCreateRequest docCreateRequest = DocumentCreateRequest.create().documents(baseDocuments);
                vectorApi.createDocuments("sample-resource-group", collectionId, docCreateRequest);

This is reproducible locally as well, with a local AICORE_SERVICE_KEY environment variable.
The service key itself does use a x509 certificate, not client secret. The same setup works with ai-sdk-java version 1.17.0.

Expected Behavior

I'd expect the document to be added correctly without an error, just like it did with 1.17.0. It does still work when downgrading to 1.17.0.

Screenshots

No response

Used Versions

Java and Maven versions:

Apache Maven 3.9.12 (848fbb4bf2d427b72bdb2471c22fced7ebd9a7a1)
Java version: 21.0.11, vendor: SAP SE, runtime: /usr/lib/jvm/sapmachine-21
Default locale: en, platform encoding: UTF-8

• SAP Cloud SDK for AI version: 1.20.0 (1.19.0 and 1.18.0 are also broken)
• Spring Boot or CAP versions:

[INFO]    com.sap.cds:cds-framework-spring-boot:jar:4.9.0:runtime -- module cds.framework.spring.boot (auto)
[INFO]    com.sap.cds:cds-starter-spring-boot-odata:jar:4.9.0:compile -- module cds.starter.spring.boot.odata (auto)
[INFO]    com.sap.cds:cds-starter-spring-boot:jar:4.9.0:compile -- module cds.starter.spring.boot (auto)
[INFO]    org.springframework.boot:spring-boot-actuator-autoconfigure:jar:3.5.15:compile -- module spring.boot.actuator.autoconfigure [auto]
[INFO]    org.springframework.boot:spring-boot-actuator:jar:3.5.15:compile -- module spring.boot.actuator [auto]
[INFO]    org.springframework.boot:spring-boot-autoconfigure:jar:3.5.15:compile -- module spring.boot.autoconfigure [auto]
[INFO]    org.springframework.boot:spring-boot-starter-actuator:jar:3.5.15:compile -- module spring.boot.starter.actuator [auto]
[INFO]    org.springframework.boot:spring-boot-starter-aop:jar:3.5.15:compile -- module spring.boot.starter.aop [auto]
[INFO]    org.springframework.boot:spring-boot-starter-jdbc:jar:3.5.15:compile -- module spring.boot.starter.jdbc [auto]
[INFO]    org.springframework.boot:spring-boot-starter-json:jar:3.5.15:compile -- module spring.boot.starter.json [auto]
[INFO]    org.springframework.boot:spring-boot-starter-logging:jar:3.5.15:compile -- module spring.boot.starter.logging [auto]
[INFO]    org.springframework.boot:spring-boot-starter-reactor-netty:jar:3.5.15:compile -- module spring.boot.starter.reactor.netty [auto]
[INFO]    org.springframework.boot:spring-boot-starter-security:jar:3.5.15:compile -- module spring.boot.starter.security [auto]
[INFO]    org.springframework.boot:spring-boot-starter-test:jar:3.5.15:test -- module spring.boot.starter.test [auto]
[INFO]    org.springframework.boot:spring-boot-starter-tomcat:jar:3.5.15:compile -- module spring.boot.starter.tomcat [auto]
[INFO]    org.springframework.boot:spring-boot-starter-validation:jar:3.5.15:compile -- module spring.boot.starter.validation [auto]
[INFO]    org.springframework.boot:spring-boot-starter-web:jar:3.5.15:compile -- module spring.boot.starter.web [auto]
[INFO]    org.springframework.boot:spring-boot-starter-webflux:jar:3.5.15:compile -- module spring.boot.starter.webflux [auto]
[INFO]    org.springframework.boot:spring-boot-starter:jar:3.5.15:compile -- module spring.boot.starter [auto]
[INFO]    org.springframework.boot:spring-boot-test-autoconfigure:jar:3.5.15:test -- module spring.boot.test.autoconfigure [auto]
[INFO]    org.springframework.boot:spring-boot-test:jar:3.5.15:test -- module spring.boot.test [auto]
[INFO]    org.springframework.boot:spring-boot:jar:3.5.15:compile -- module spring.boot [auto]

(created by running mvn dependency:list | grep spring-boot | uniq | sort)

Code Examples

List<BaseDocument> baseDocuments = List.of(BaseDocument.create()
                      .chunks(List.of(TextOnlyBaseChunkCreate.create().content("sample content")
                              .metadata(List.of(VectorKeyValueListPair.create().key("key").value("value")))))
                      .metadata(List.of(VectorDocumentKeyValueListPair.create().key("key").value("value"))));

              // Create documents by calling vector api
              DocumentCreateRequest docCreateRequest = DocumentCreateRequest.create().documents(baseDocuments);
              vectorApi.createDocuments("sample-resource-group", collectionId, docCreateRequest);

Stack Trace

com.sap.cloud.sdk.services.openapi.apache.core.OpenApiRequestException: java.io.IOException: Failed to process response: HTTP/1.1 500 Internal Server Error
        at com.sap.cloud.sdk.services.openapi.apache.apiclient.ApiClient.invokeAPI(ApiClient.java:511)
        at com.sap.ai.sdk.grounding.client.VectorApi.createDocuments(VectorApi.java:220)
        <---------- internal stack trace content left out ---------------->
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:1272)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1797)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:973)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:491)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: Failed to process response: HTTP/1.1 500 Internal Server Error
        at com.sap.cloud.sdk.services.openapi.apache.apiclient.DefaultApiResponseHandler.handleResponse(DefaultApiResponseHandler.java:86)
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:247)
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188)
        at com.sap.cloud.sdk.services.openapi.apache.apiclient.ApiClient.invokeAPI(ApiClient.java:508)
        ... 154 more
Caused by: com.sap.cloud.sdk.services.openapi.apache.core.OpenApiRequestException: HTTP/1.1 500 Internal Server Error
        at com.sap.cloud.sdk.services.openapi.apache.apiclient.DefaultApiResponseHandler.processResponse(DefaultApiResponseHandler.java:127)
        at com.sap.cloud.sdk.services.openapi.apache.apiclient.DefaultApiResponseHandler.handleResponse(DefaultApiResponseHandler.java:82)
        ... 157 more

Log File

DEBUG <redacted> org.apache.hc.client5.http.wire          : http-outgoing-1 << "{"error": {"code": "500", "message": "Internal Server Error", "requestId": "<redacted>", "details": [{"code": "500", "message": "Internal server error when calling document grounding api."}]}}"

Affected Development Phase

Production

Impact

Impaired

Timeline

No response

[Jonas-Isr]

Hi @mweiler-sap,

thanks for reaching out! This is a (somewhat) known issue with the grounding service. In their official API spec they have marked the matchMode property of the VectorDocumentKeyValueListPair as nullable (you can find a copy of their spec here). Thus, we send that value as null if it is not set in compliance with the spec in our generated code (which generates directly from that spec). Unfortunately, this raises an internal server error on their side.

The workaround is to set the value to ANY (or another value that makes sense for you). The below version of your code snippet works in my tests. You can also see this in our sample code.

List<BaseDocument> baseDocuments = List.of(BaseDocument.create()
                      .chunks(List.of(TextOnlyBaseChunkCreate.create().content("sample content")
                              .metadata(List.of(VectorKeyValueListPair.create().key("key").value("value")))))
                              .metadata(List.of(
                                    VectorDocumentKeyValueListPair.create().key("key").value("value").matchMode(FilterMatchModeEnum.ANY))));

// Create documents by calling vector api
DocumentCreateRequest docCreateRequest = DocumentCreateRequest.create().documents(baseDocuments);
vectorApi.createDocuments("sample-resource-group", collectionId, docCreateRequest);

This happens only from version 1.18.0 of our SDK onwards since then we applied the change in the grounding service spec.

I hope this helps!
Best,
Jonas