Skip to content

Commit 91ad5d5

Browse files
authored
ADD FoundationDB storage step1 (pubkey#3954)
* ADD FoundationDB storage step1 * FIX(foundationdb) lwt based indexes * ADD docs for foundationDB * FIX installation of foundationdb * FIX install * FIX build * FIX foundationdb tests * FIX scripts * ADD(foundationdb) attachments support * FIX build * FIX build again * FIX browser tests * FIX hacky lol
1 parent 411ccc8 commit 91ad5d5

25 files changed

+951
-29
lines changed

.github/workflows/main.yml

+37
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,43 @@ jobs:
258258
with:
259259
run: npm run test:browser:ci:dexie
260260

261+
storage-foundationdb:
262+
runs-on: ubuntu-20.04
263+
steps:
264+
- uses: actions/checkout@v2
265+
- name: Set node version
266+
uses: actions/setup-node@v3
267+
with:
268+
node-version-file: '.nvmrc'
269+
270+
- name: Reuse npm cache folder
271+
uses: actions/cache@v2
272+
env:
273+
cache-name: cache-node-modules
274+
with:
275+
path: |
276+
~/.npm
277+
./node_modules
278+
key: ${{ runner.os }}-npm-storage-foundationdb-x7-${{ hashFiles('**/package.json') }}
279+
restore-keys: |
280+
${{ runner.os }}-npm-storage-foundationdb-x7-
281+
282+
- name: install FoundationDB client and server
283+
working-directory: ./scripts
284+
run: sh install-foundationdb.sh
285+
286+
- name: install npm dependencies
287+
run: npm install
288+
289+
- run: npm install [email protected]
290+
291+
- name: build
292+
run: npm run build
293+
294+
- run: npm run test:node:foundationdb
295+
- run: npm run test:fast:foundationdb
296+
297+
261298
angular:
262299
runs-on: ubuntu-20.04
263300
steps:

CHANGELOG.md

+7-6
Original file line numberDiff line numberDiff line change
@@ -9,14 +9,14 @@
99

1010
- RENAMED the `ajv-validate` plugin to `validate-ajv` to be in equal with the other validation plugins.
1111
- The `is-my-json-valid` validation is no longer supported until [this bug](https://github.com/mafintosh/is-my-json-valid/pull/192) is fixed.
12-
- REFACTORED the [schema validation plugins](https://rxdb.info/schema-validation.html), they are no longer plugins but now they get wrapped around any other RxStorage.
13-
- It allows us to run the validation inside of a [Worker RxStorage](./rx-storage-worker.md) instead of running it in the main JavaScript process.
12+
- REFACTORED the [schema validation plugins](./docs-src/schema-validation.md), they are no longer plugins but now they get wrapped around any other RxStorage.
13+
- It allows us to run the validation inside of a [Worker RxStorage](./docs-src/rx-storage-worker.md) instead of running it in the main JavaScript process.
1414
- It allows us to configure which `RxDatabase` instance must use the validation and which does not. In production it often makes sense to validate user data, but you might not need the validation for data that is only replicated from the backend.
15-
- REFACTORED the [key compression plugin](https://rxdb.info/key-compression.html), it is no longer a plugin but now a wrapper around any other RxStorage.
16-
- It allows to run the key-comresion inside of a [Worker RxStorage](./rx-storage-worker.md) instead of running it in the main JavaScript process.
15+
- REFACTORED the [key compression plugin](./docs-src/key-compression.md), it is no longer a plugin but now a wrapper around any other RxStorage.
16+
- It allows to run the key-comresion inside of a [Worker RxStorage](./docs-src/rx-storage-worker.md) instead of running it in the main JavaScript process.
1717

1818
- REFACTORED the encryption plugin, it is no longer a plugin but now a wrapper around any other RxStorage.
19-
- It allows to run the encryption inside of a [Worker RxStorage](./rx-storage-worker.md) instead of running it in the main JavaScript process.
19+
- It allows to run the encryption inside of a [Worker RxStorage](./docs-src/rx-storage-worker.md) instead of running it in the main JavaScript process.
2020
- It allows do use asynchronous crypto function like [WebCrypto](https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API)
2121
- Store the password hash in the same write request as the database token to improve performance.
2222

@@ -44,7 +44,8 @@
4444
- FIXED multiple problems with encoding attachments data. We now use the `js-base64` library which properly handles utf-8/binary/ascii transformations.
4545

4646
- RENAMED the `server` plugin is now called `server-couchdb` and `RxDatabase.server()` is now `RxDatabase.serverCouchDB()`
47-
- ADDED the [websocket replication plugin](../replication-websocket.md)
47+
- ADDED the [websocket replication plugin](./docs-src/replication-websocket.md)
48+
- ADDED the [FoundationDB RxStorage](./docs-src/rx-storage-foundationdb.md)
4849

4950
- FIX `couchdb-server` plugin missed out events from the replication.
5051

docs-src/SUMMARY.md

+1
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@
121121
* [RxStorage Memory](./rx-storage-memory.md)
122122
* [RxStorage IndexedDB](./rx-storage-indexeddb.md)
123123
* [RxStorage SQLite](./rx-storage-sqlite.md)
124+
* [RxStorage FoundationDB](./rx-storage-foundationdb.md)
124125
* [RxStorage Worker](./rx-storage-worker.md)
125126
* [RxStorage Memory Synced](./rx-storage-memory-synced.md)
126127
* [RxStorage Sharding](./rx-storage-sharding.md)

docs-src/releases/13.0.0.md

+1
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ When used with Node.js, RxDB now requires Node.js version `18.0.0` or higher.
8989
## New Features
9090

9191
- ADDED the [websocket replication plugin](../replication-websocket.md)
92+
- ADDED the [FoundationDB RxStorage](../rx-storage-foundationdb.md)
9293

9394
## Migration to the new version
9495

docs-src/rx-storage-foundationdb.md

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# RxStorage FoundationDB (beta)
2+
3+
To use RxDB on the server side, the [FoundationDB](https://www.foundationdb.org/) [RxStorage](./rx-storage.md) provides a way of having a secure, fault-tolerant and performant storage.
4+
5+
## Installation
6+
7+
- Install the [FoundationDB client cli](https://apple.github.io/foundationdb/getting-started-linux.html) which is used to communicate with the FoundationDB cluster.
8+
- Install the [FoundationDB node bindings npm module](https://www.npmjs.com/package/foundationdb) via `npm install foundationdb --save`. If the latest version does not work for you, you should use the same version as stated in the `storage-foundationdb` job of the RxDB CI `main.yml`.
9+
10+
11+
## Usage
12+
13+
```typescript
14+
import {
15+
createRxDatabase
16+
} from 'rxdb';
17+
import {
18+
getRxStorageFoundationDB
19+
} from 'rxdb/plugins/foundationdb';
20+
21+
const db = await createRxDatabase({
22+
name: 'exampledb',
23+
storage: getRxStorageFoundationDB({
24+
/**
25+
* Version of the API of the FoundationDB cluster..
26+
* FoundationDB is backwards compatible across a wide range of versions,
27+
* so you have to specify the api version.
28+
* If in doubt, set it to 620.
29+
*/
30+
apiVersion: 620,
31+
/**
32+
* Path to the FoundationDB cluster file.
33+
* (optional)
34+
* If in doubt, leave this empty to use the default location.
35+
*/
36+
clusterFile: '/path/to/fdb.cluster',
37+
/**
38+
* Amount of documents to be fetched in batch requests.
39+
* You can change this to improve performance depending on
40+
* your database access patterns.
41+
* (optional)
42+
* [default=50]
43+
*/
44+
batchSize: 50
45+
})
46+
});
47+
```
48+
49+
## Multi Instance
50+
51+
Because FoundationDB does not offer a [changestream](https://forums.foundationdb.org/t/streaming-data-out-of-foundationdb/683/2), it is not possible to use the same cluster from more then one Node.js process at the same time. For example you cannot spin up multiple servers with RxDB databases that all use the same cluster. There might be workarounds to create something like a FoundationDB changestream and you can make a Pull Request if you need that feature.

docs-src/rx-storage.md

+4
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@ The IndexedDB `RxStorage` is based on plain IndexedDB. This has the best perform
3030

3131
The SQLite storage has the best performance when RxDB is used on **Node.js**, **Electron**, **React Native**, **Cordova** or **Capacitor**. [Read more](./rx-storage-sqlite.md)
3232

33+
### FoundationDB
34+
35+
To use RxDB on the server side, the FoundationDB RxStorage provides a way of having a secure, fault-tolerant and performant storage. [Read more](./rx-storage-foundationdb.md)
36+
3337
### Worker
3438

3539
The worker RxStorage is a wrapper around any other RxStorage which allows to run the storage in a WebWorker (in browsers) or a Worker Thread (in node.js). By doing so, you can take CPU load from the main process and move it into the worker's process which can improve the percieved performance of your application. [Read more](./rx-storage-worker.md)

package.json

+3-1
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
"test:fast:lokijs": "npm run transpile && cross-env DEFAULT_STORAGE=lokijs NODE_ENV=fast mocha --config ./config/.mocharc.js ./test_tmp/unit.test.js",
4444
"test:fast:dexie": "npm run transpile && cross-env DEFAULT_STORAGE=dexie NODE_ENV=fast mocha --config ./config/.mocharc.js ./test_tmp/unit.test.js",
4545
"test:fast:dexie-worker": "npm run transpile && npm run build:workers && rimraf -rf pouch__all_dbs__ && cross-env DEFAULT_STORAGE=dexie-worker NODE_ENV=fast mocha --config ./config/.mocharc.js ./test_tmp/unit.test.js",
46+
"test:fast:foundationdb": "npm run transpile && cross-env DEFAULT_STORAGE=foundationdb NODE_ENV=fast mocha --config ./config/.mocharc.js ./test_tmp/unit.test.js",
4647
"// test:fast:loop": "runs tests in the fast-mode in a loop. Use this to debug tests that only fail sometimes",
4748
"test:fast:loop": "npm run test:fast && npm run test:fast:loop",
4849
"test:fast:loop:pouchdb": "npm run test:fast:pouchdb && npm run test:fast:loop:pouchdb",
@@ -57,6 +58,7 @@
5758
"test:node:lokijs": "npm run transpile && cross-env DEFAULT_STORAGE=lokijs mocha --expose-gc --config ./config/.mocharc.js ./test_tmp/unit.test.js",
5859
"test:node:dexie-worker": "npm run transpile && npm run build:workers && cross-env DEFAULT_STORAGE=dexie-worker mocha --expose-gc --config ./config/.mocharc.js ./test_tmp/unit.test.js",
5960
"test:node:dexie": "npm run transpile && cross-env DEFAULT_STORAGE=dexie mocha --expose-gc --config ./config/.mocharc.js ./test_tmp/unit.test.js",
61+
"test:node:foundationdb": "npm run transpile && cross-env DEFAULT_STORAGE=foundationdb mocha --expose-gc --config ./config/.mocharc.js ./test_tmp/unit.test.js",
6062
"test:node:pouchdb:loop": "npm run test:node:pouchdb && npm run test:node:pouchdb:loop",
6163
"test:node:lokijs:loop": "npm run test:node:lokijs && npm run test:node:lokijs:loop",
6264
"test:node:dexie:loop": "npm run test:node:dexie && npm run test:node:dexie:loop",
@@ -275,4 +277,4 @@
275277
"webpack-bundle-analyzer": "4.5.0",
276278
"webpack-cli": "4.10.0"
277279
}
278-
}
280+
}

plugins/foundationdb/package.json

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"name": "rxdb-plugin-foundationdb",
3+
"main": "../../dist/lib/plugins/foundationdb/index.js",
4+
"jsnext:main": "../../dist/es/plugins/foundationdb/index.js",
5+
"module": "../../dist/es/plugins/foundationdb/index.js",
6+
"types": "../../dist/types/plugins/foundationdb/index.d.ts",
7+
"sideEffects": false
8+
}

scripts/install-foundationdb.sh

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/bash
2+
set -e
3+
4+
echo "# installing FoundationDB client:"
5+
wget -O foundationdb-client.deb https://github.com/apple/foundationdb/releases/download/6.3.23/foundationdb-clients_6.3.23-1_amd64.deb
6+
sudo dpkg -i foundationdb-client.deb
7+
rm foundationdb-client.deb
8+
9+
10+
echo "# installing FoundationDB server:"
11+
wget -O foundationdb-server.deb https://github.com/apple/foundationdb/releases/download/6.3.23/foundationdb-server_6.3.23-1_amd64.deb
12+
sudo dpkg -i foundationdb-server.deb
13+
rm foundationdb-server.deb
14+

src/custom-index.ts

+5-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@ import type {
1010
RxDocumentData,
1111
RxJsonSchema
1212
} from './types';
13-
import { ensureNotFalsy, objectPathMonad, ObjectPathMonadFunction } from './util';
13+
import {
14+
ensureNotFalsy,
15+
objectPathMonad,
16+
ObjectPathMonadFunction
17+
} from './util';
1418
import { INDEX_MAX } from './query-planner';
1519

1620

src/plugins/attachments.ts

-1
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,6 @@ export class RxAttachment {
127127
this.doc.primary,
128128
this.id
129129
);
130-
console.dir(plainDataBase64);
131130
const ret = await blobBufferUtil.createBlobBufferFromBase64(
132131
plainDataBase64,
133132
this.type as any
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
export function getFoundationDBIndexName(index: string[]): string {
2+
return index.join('|');
3+
}
4+
export const CLEANUP_INDEX: string[] = ['_deleted', '_meta.lwt'];
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
import {
2+
getStartIndexStringFromLowerBound,
3+
getStartIndexStringFromUpperBound
4+
} from '../../custom-index';
5+
import type {
6+
RxDocumentData,
7+
RxStorageQueryResult
8+
} from '../../types';
9+
import { ensureNotFalsy } from '../../util';
10+
import { RxStorageDexieStatics } from '../dexie';
11+
import { getFoundationDBIndexName } from './foundationdb-helpers';
12+
import type {
13+
FoundationDBPreparedQuery
14+
} from './foundationdb-types';
15+
import { RxStorageInstanceFoundationDB } from './rx-storage-instance-foundationdb';
16+
17+
export async function queryFoundationDB<RxDocType>(
18+
instance: RxStorageInstanceFoundationDB<RxDocType>,
19+
preparedQuery: FoundationDBPreparedQuery<RxDocType>
20+
): Promise<RxStorageQueryResult<RxDocType>> {
21+
const queryPlan = preparedQuery.queryPlan;
22+
const query = preparedQuery.query;
23+
const skip = query.skip ? query.skip : 0;
24+
const limit = query.limit ? query.limit : Infinity;
25+
const skipPlusLimit = skip + limit;
26+
const queryPlanFields: string[] = queryPlan.index;
27+
const mustManuallyResort = !queryPlan.sortFieldsSameAsIndexFields;
28+
29+
const queryMatcher = RxStorageDexieStatics.getQueryMatcher(
30+
instance.schema,
31+
preparedQuery
32+
);
33+
const sortComparator = RxStorageDexieStatics.getSortComparator(instance.schema, preparedQuery);
34+
const dbs = await instance.internals.dbsPromise;
35+
36+
37+
const indexForName = queryPlanFields.slice(0);
38+
indexForName.unshift('_deleted');
39+
const indexName = getFoundationDBIndexName(indexForName);
40+
const indexDB = ensureNotFalsy(dbs.indexes[indexName]).db;
41+
42+
let lowerBound: any[] = queryPlan.startKeys;
43+
lowerBound = [false].concat(lowerBound);
44+
const lowerBoundString = getStartIndexStringFromLowerBound(
45+
instance.schema,
46+
indexForName,
47+
lowerBound
48+
);
49+
50+
let upperBound: any[] = queryPlan.endKeys;
51+
upperBound = [false].concat(upperBound);
52+
const upperBoundString = getStartIndexStringFromUpperBound(
53+
instance.schema,
54+
indexForName,
55+
upperBound
56+
);
57+
let result = await dbs.root.doTransaction(async (tx: any) => {
58+
const innerResult: RxDocumentData<RxDocType>[] = [];
59+
const indexTx = tx.at(indexDB.subspace);
60+
const mainTx = tx.at(dbs.main.subspace);
61+
62+
const range = indexTx.getRangeBatch(
63+
lowerBoundString,
64+
upperBoundString,
65+
{
66+
// TODO these options seem to be broken in the foundationdb node bindings
67+
// limit: instance.settings.batchSize,
68+
// streamingMode: StreamingMode.Exact
69+
}
70+
);
71+
let done = false;
72+
while (!done) {
73+
const next = await range.next();
74+
if (next.done) {
75+
done = true;
76+
break;
77+
}
78+
const docIds = next.value.map((row: string[]) => row[1]);
79+
const docsData: RxDocumentData<RxDocType>[] = await Promise.all(docIds.map((docId: string) => mainTx.get(docId)));
80+
docsData.forEach((docData) => {
81+
if (!done) {
82+
if (
83+
queryMatcher(docData)
84+
) {
85+
innerResult.push(docData);
86+
}
87+
}
88+
if (
89+
!mustManuallyResort &&
90+
innerResult.length === skipPlusLimit
91+
) {
92+
done = true;
93+
range.return();
94+
}
95+
});
96+
}
97+
return innerResult;
98+
});
99+
if (mustManuallyResort) {
100+
result = result.sort(sortComparator);
101+
}
102+
103+
// apply skip and limit boundaries.
104+
result = result.slice(skip, skipPlusLimit);
105+
106+
return {
107+
documents: result
108+
};
109+
}

0 commit comments

Comments
 (0)