Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial F.A.R.R related implementations #106

Merged
merged 36 commits into from
Apr 6, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
4257397
initial implementatino of new cursors
toptobes Mar 11, 2025
3ff014d
initial findAndRerank implementation
toptobes Mar 11, 2025
8d71659
run premerge script
toptobes Mar 11, 2025
eb784ae
oops
toptobes Mar 11, 2025
acab191
minor tweaks
toptobes Mar 11, 2025
27b9046
some cursor code deduplication
toptobes Mar 12, 2025
95ff780
add types for reranking & lexical
toptobes Mar 12, 2025
ae26080
add createtextindex
toptobes Mar 12, 2025
c223afe
fix type error
toptobes Mar 12, 2025
5a45a5a
add rerankresult
toptobes Mar 12, 2025
e5bddc3
initial implementation of batch overhauls mvp
toptobes Mar 12, 2025
23021de
Love coding? here's the secret reason why
toptobes Mar 12, 2025
7279ddc
tiny changes
toptobes Mar 12, 2025
256e2e1
run build report
toptobes Mar 12, 2025
a0127cc
more tiny changes
toptobes Mar 12, 2025
af22083
tiny changes
toptobes Mar 12, 2025
7561dc3
rerankField => rerankOn
toptobes Mar 12, 2025
774727c
rip out listIndexes
toptobes Mar 12, 2025
e6fcbb8
minro test fixes
toptobes Mar 13, 2025
6d0d91b
add field escaping utility functions
toptobes Mar 13, 2025
d1b9c0e
patched tests to work on dataapi main branch
toptobes Mar 15, 2025
2aef1dd
Logging restructuring (#104)
toptobes Mar 15, 2025
13ff417
Pipeline goes brrrrrrr
toptobes Mar 15, 2025
cbe23ef
strip trailing slash in db endpoint
toptobes Mar 16, 2025
4603f00
fix some test lib bug
toptobes Mar 16, 2025
ccb46a0
findRerankingProviders impl
toptobes Mar 16, 2025
5d50c25
run premerge script
toptobes Mar 16, 2025
b34f65c
start implementing new headers provdier + restructuring
toptobes Mar 17, 2025
785b8d2
more such work
toptobes Mar 17, 2025
92aab02
minor updates to timeout error
toptobes Mar 18, 2025
ee80ee2
run build report
toptobes Mar 18, 2025
75e5819
run build report + fix some missing exports
toptobes Mar 18, 2025
9a19dc3
bad things happen when you forget about that one little change you ma…
toptobes Mar 18, 2025
ea8b4bc
make consumeBuffer increase consumed
toptobes Mar 18, 2025
098b10b
made rerankQuery and includeScores configurable options
toptobes Mar 25, 2025
809e7af
fix issue from merging
toptobes Apr 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions api-extractor.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -402,10 +402,10 @@
// "addToApiReportFile": false
},

// "ae-internal-missing-underscore": {
// "logLevel": "none",
// "addToApiReportFile": false
// }
"ae-internal-missing-underscore": {
"logLevel": "none",
"addToApiReportFile": false
},

"ae-unresolved-link": {
"logLevel": "none",
Expand Down
805 changes: 567 additions & 238 deletions etc/astra-db-ts.api.md

Large diffs are not rendered by default.

Binary file modified examples/astra-db-ts.tgz
Binary file not shown.
2 changes: 1 addition & 1 deletion examples/browser/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/cloudflare-workers/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/customize-http/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/logging/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/nextjs/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/non-astra-backends/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/serdes/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/using-http2/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion scripts/repl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ node -i -e "
let table = db.table('$default_table_name');

if (process.env.LOG_ALL_TO_STDOUT) {
for (const event of ['commandSucceeded', 'adminCommandSucceeded', 'commandFailed', 'adminCommandFailed']) {
for (const event of ['commandSucceeded', 'adminCommandSucceeded', 'commandFailed', 'adminCommandFailed', /.*Warning/]) {
client.on(event, (e) => console.dir(e, { depth: null }));
}
}
Expand Down
16 changes: 8 additions & 8 deletions src/administration/astra-admin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import { buildAstraDatabaseAdminInfo } from '@/src/administration/utils.js';
import { DEFAULT_DEVOPS_API_ENDPOINTS, DEFAULT_KEYSPACE, HttpMethods } from '@/src/lib/api/constants.js';
import { DevOpsAPIHttpClient } from '@/src/lib/api/clients/devops-api-http-client.js';
import type { OpaqueHttpClient, WithTimeout } from '@/src/lib/index.js';
import { HierarchicalEmitter, TokenProvider } from '@/src/lib/index.js';
import { HierarchicalLogger, TokenProvider } from '@/src/lib/index.js';
import type { AstraDbAdminInfo } from '@/src/administration/types/admin/database-info.js';
import { buildAstraEndpoint } from '@/src/lib/utils.js';
import type { DbOptions } from '@/src/client/index.js';
Expand Down Expand Up @@ -64,7 +64,7 @@ import type { AdminCommandEventMap } from '@/src/administration/events.js';
*
* @public
*/
export class AstraAdmin extends HierarchicalEmitter<AdminCommandEventMap> {
export class AstraAdmin extends HierarchicalLogger<AdminCommandEventMap> {
readonly #defaultOpts: ParsedRootClientOpts;
readonly #httpClient: DevOpsAPIHttpClient;
readonly #environment: 'dev' | 'test' | 'prod';
Expand All @@ -75,9 +75,7 @@ export class AstraAdmin extends HierarchicalEmitter<AdminCommandEventMap> {
* @internal
*/
constructor(rootOpts: ParsedRootClientOpts, adminOpts: ParsedAdminOptions) {
super(rootOpts.client);

this.#defaultOpts = {
const defaultOpts = {
...rootOpts,
adminOptions: AdminOptsHandler.concat([rootOpts.adminOptions, adminOpts]),
dbOptions: {
Expand All @@ -86,16 +84,18 @@ export class AstraAdmin extends HierarchicalEmitter<AdminCommandEventMap> {
},
};

super(rootOpts.client, defaultOpts.adminOptions.logging);

this.#defaultOpts = defaultOpts;
this.#environment = this.#defaultOpts.adminOptions.astraEnv ?? 'prod';

this.#httpClient = new DevOpsAPIHttpClient({
logging: this.#defaultOpts.adminOptions.logging,
baseUrl: this.#defaultOpts.adminOptions.endpointUrl ?? DEFAULT_DEVOPS_API_ENDPOINTS[this.#environment],
emitter: this,
logger: this,
fetchCtx: rootOpts.fetchCtx,
caller: rootOpts.caller,
tokenProvider: this.#defaultOpts.adminOptions.adminToken,
additionalHeaders: this.#defaultOpts.adminOptions.additionalHeaders,
additionalHeaders: this.#defaultOpts.additionalHeaders,
timeoutDefaults: Timeouts.cfg.concat([rootOpts.adminOptions.timeoutDefaults, this.#defaultOpts.adminOptions.timeoutDefaults]),
});

Expand Down
45 changes: 13 additions & 32 deletions src/administration/astra-db-admin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,13 @@ import { DbAdmin } from '@/src/administration/db-admin.js';
import type { OpaqueHttpClient, WithTimeout } from '@/src/lib/index.js';
import { TokenProvider } from '@/src/lib/index.js';
import { buildAstraDatabaseAdminInfo, extractAstraEnvironment } from '@/src/administration/utils.js';
import type { FindEmbeddingProvidersResult } from '@/src/administration/types/db-admin/find-embedding-providers.js';
import { DEFAULT_DEVOPS_API_ENDPOINTS, HttpMethods } from '@/src/lib/api/constants.js';
import { DevOpsAPIHttpClient } from '@/src/lib/api/clients/devops-api-http-client.js';
import type { Db } from '@/src/db/index.js';
import { $CustomInspect } from '@/src/lib/constants.js';
import type { AstraDbAdminInfo } from '@/src/administration/types/admin/database-info.js';
import { Logger } from '@/src/lib/logging/logger.js';
import type { TimeoutManager} from '@/src/lib/api/timeouts/timeouts.js';
import { InternalLogger } from '@/src/lib/logging/internal-logger.js';
import type { TimeoutManager } from '@/src/lib/api/timeouts/timeouts.js';
import { Timeouts } from '@/src/lib/api/timeouts/timeouts.js';
import type { DataAPIHttpClient } from '@/src/lib/api/clients/index.js';
import type { ParsedAdminOptions } from '@/src/client/opts-handlers/admin-opts-handler.js';
Expand Down Expand Up @@ -77,18 +76,18 @@ export class AstraDbAdmin extends DbAdmin {
* @internal
*/
constructor(db: Db, rootOpts: ParsedRootClientOpts, adminOpts: ParsedAdminOptions, dbToken: ParsedTokenProvider, endpoint: string) {
super(rootOpts.client);
const loggingConfig = InternalLogger.cfg.concat([rootOpts.adminOptions.logging, adminOpts.logging]);
super(rootOpts.client, loggingConfig);

this.#environment = adminOpts?.astraEnv ?? rootOpts.adminOptions.astraEnv ?? extractAstraEnvironment(endpoint);

this.#httpClient = new DevOpsAPIHttpClient({
baseUrl: DEFAULT_DEVOPS_API_ENDPOINTS[this.#environment],
logging: Logger.cfg.concat([rootOpts.adminOptions.logging, adminOpts.logging]),
fetchCtx: rootOpts.fetchCtx,
emitter: this,
logger: this,
caller: rootOpts.caller,
tokenProvider: TokenProvider.opts.concat([dbToken, rootOpts.adminOptions.adminToken, adminOpts.adminToken]),
additionalHeaders: { ...rootOpts.adminOptions.additionalHeaders, ...adminOpts?.additionalHeaders },
additionalHeaders: rootOpts.additionalHeaders,
timeoutDefaults: Timeouts.cfg.concat([rootOpts.adminOptions.timeoutDefaults, adminOpts.timeoutDefaults]),
});

Expand Down Expand Up @@ -130,31 +129,6 @@ export class AstraDbAdmin extends DbAdmin {
return this.#db;
}

/**
* Returns detailed information about the availability and usage of the vectorize embedding providers available on the
* current database (may vary based on cloud provider & region).
*
* @example
* ```typescript
* const { embeddingProviders } = await dbAdmin.findEmbeddingProviders();
*
* // ['text-embedding-3-small', 'text-embedding-3-large', 'text-embedding-ada-002']
* console.log(embeddingProviders['openai'].models.map(m => m.name));
* ```
*
* @param options - The options for the timeout of the operation.
*
* @returns The available embedding providers.
*/
public override async findEmbeddingProviders(options?: WithTimeout<'databaseAdminTimeoutMs'>): Promise<FindEmbeddingProvidersResult> {
const resp = await this.#dataApiHttpClient.executeCommand({ findEmbeddingProviders: {} }, {
timeoutManager: this.#httpClient.tm.single('databaseAdminTimeoutMs', options),
methodName: 'dbAdmin.findEmbeddingProviders',
keyspace: null,
});
return resp.status as FindEmbeddingProvidersResult;
}

/**
* Fetches the complete information about the database, such as the database name, IDs, region, status, actions, and
* other metadata.
Expand Down Expand Up @@ -346,4 +320,11 @@ export class AstraDbAdmin extends DbAdmin {

return buildAstraDatabaseAdminInfo(resp.data!, this.#environment);
}

/**
* @internal
*/
protected override _getDataAPIHttpClient(): DataAPIHttpClient<'admin'> {
return this.#dataApiHttpClient;
}
}
40 changes: 12 additions & 28 deletions src/administration/data-api-db-admin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,13 @@
import type { DataAPICreateKeyspaceOptions } from '@/src/administration/types/index.js';
import { DbAdmin } from '@/src/administration/db-admin.js';
import type { OpaqueHttpClient, WithTimeout } from '@/src/lib/index.js';
import type { FindEmbeddingProvidersResult } from '@/src/administration/types/db-admin/find-embedding-providers.js';
import type { DataAPIHttpClient } from '@/src/lib/api/clients/data-api-http-client.js';
import type { Db } from '@/src/db/index.js';
import { $CustomInspect } from '@/src/lib/constants.js';
import type { ParsedAdminOptions } from '@/src/client/opts-handlers/admin-opts-handler.js';
import type { DataAPIClient } from '@/src/client/data-api-client.js';
import { InternalLogger } from '@/src/lib/logging/internal-logger.js';
import type { ParsedRootClientOpts } from '@/src/client/opts-handlers/root-opts-handler.js';

/**
* An administrative class for managing non-Astra databases, including creating, listing, and deleting keyspaces.
Expand Down Expand Up @@ -65,8 +66,9 @@ export class DataAPIDbAdmin extends DbAdmin {
*
* @internal
*/
constructor(db: Db, client: DataAPIClient, httpClient: DataAPIHttpClient, adminOpts: ParsedAdminOptions) {
super(client);
constructor(db: Db, client: DataAPIClient, httpClient: DataAPIHttpClient, rootOpts: ParsedRootClientOpts, adminOpts: ParsedAdminOptions) {
const loggingConfig = InternalLogger.cfg.concat([rootOpts.dbOptions.logging, adminOpts.logging]);
super(client, loggingConfig);

this.#httpClient = httpClient.forDbAdmin(this, adminOpts);
this.#db = db;
Expand Down Expand Up @@ -97,31 +99,6 @@ export class DataAPIDbAdmin extends DbAdmin {
return this.#db;
}

/**
* Returns detailed information about the availability and usage of the vectorize embedding providers available on the
* current database (may vary based on cloud provider & region).
*
* @example
* ```typescript
* const { embeddingProviders } = await dbAdmin.findEmbeddingProviders();
*
* // ['text-embedding-3-small', 'text-embedding-3-large', 'text-embedding-ada-002']
* console.log(embeddingProviders['openai'].models.map(m => m.name));
* ```
*
* @param options - The options for the timeout of the operation.
*
* @returns The available embedding providers.
*/
public override async findEmbeddingProviders(options?: WithTimeout<'databaseAdminTimeoutMs'>): Promise<FindEmbeddingProvidersResult> {
const resp = await this.#httpClient.executeCommand({ findEmbeddingProviders: {} }, {
timeoutManager: this.#httpClient.tm.single('databaseAdminTimeoutMs', options),
methodName: 'dbAdmin.findEmbeddingProviders',
keyspace: null,
});
return resp.status as FindEmbeddingProvidersResult;
}

/**
* Lists the keyspaces in the database.
*
Expand Down Expand Up @@ -226,4 +203,11 @@ export class DataAPIDbAdmin extends DbAdmin {
public get _httpClient(): OpaqueHttpClient {
return this.#httpClient;
}

/**
* @internal
*/
protected override _getDataAPIHttpClient(): DataAPIHttpClient<'admin'> {
return this.#httpClient;
}
}
Loading