Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
8c1a9b1
add api token picker to ChatModelEditView
Feb 20, 2025
5d79140
add Bearer header field
Feb 20, 2025
6a90966
optionally add api key to header
Feb 20, 2025
2bed2b6
Merge pull request #625 from tmpit/ollama_api_token
intitni Feb 21, 2025
94bfdd5
Add API key field for Ollama embedding API
intitni Feb 21, 2025
be47c9a
Add custom header filed for Ollama APIs
intitni Feb 21, 2025
f1cb526
Update model format picker styles
intitni Feb 21, 2025
4298221
Add id to references
intitni Feb 23, 2025
8a0b121
Support reasoning content from DeepSeek
intitni Feb 23, 2025
316d91d
Update
intitni Feb 23, 2025
a7a34fa
Fix embedding empty array
intitni Feb 23, 2025
88ce86f
Update UI
intitni Feb 23, 2025
e4b2336
Add request modifier
intitni Feb 24, 2025
17514e3
Add GitHub Copilot auth token fetcher
intitni Feb 24, 2025
2710000
Update name
intitni Feb 24, 2025
d8f95d2
Add method to get GitHubCopilot tokens
intitni Feb 24, 2025
e6dee6b
Add header value parser
intitni Feb 24, 2025
dca1c0d
Support joining split documents
intitni Feb 24, 2025
50270bf
Add new models
intitni Feb 24, 2025
5e8d1ec
Adjust debug UI z index
intitni Feb 24, 2025
7c2c89b
Support github copilot chat direct call
intitni Feb 24, 2025
5f07431
Support embedding with GitHub Copilot
intitni Feb 24, 2025
dc2d2fc
Fix selected content
intitni Feb 24, 2025
0051a26
Add option requiresBeginWithUserMessage
intitni Feb 24, 2025
efeafb0
Fix selected content
intitni Feb 24, 2025
addd0d5
Bump version
intitni Feb 24, 2025
730669f
Fix document merging
intitni Feb 25, 2025
1acd818
Enable NSAllowsArbitraryLoads
intitni Feb 25, 2025
1b80c9d
Fix configuration override
intitni Feb 25, 2025
8110127
Fix header parsing
intitni Feb 25, 2025
b70c21e
Merge branch 'hotfix/0.35.5'
intitni Feb 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix document merging
  • Loading branch information
intitni committed Feb 25, 2025
commit 730669ff80d69a6adc376db295733e661ceb7ebf
13 changes: 11 additions & 2 deletions Tool/Sources/LangChain/DocumentTransformer/TextSplitter.swift
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,17 @@ public extension TextSplitter {
endUTF16Offset: end
)
}.sorted(by: { $0.startUTF16Offset < $1.startUTF16Offset })
let mergedChunks = mergeSplits(textChunks)
let pageContent = mergedChunks.map(\.text).joined()
var sumChunk: TextChunk?
for chunk in textChunks {
if let current = sumChunk {
if let merged = current.merged(with: chunk, force: true) {
sumChunk = merged
}
} else {
sumChunk = chunk
}
}
let pageContent = sumChunk?.text ?? ""
var metadata = documents.first?.metadata ?? [String: JSONValue]()
metadata["startUTF16Offset"] = nil
metadata["endUTF16Offset"] = nil
Expand Down