Queries I Found Useful in My First Adventures with Firestore

16/05/2021 — JavaScript, Software Development — 4 min read

After a pretty successful time building a prototype of Reciprocal.dev we decided to start building a beta that added user accounts and the ability for users to create their own user journey maps with the tool.

I decided to go with Firebase for both the authentication and data storage solution as the interoperability of both solutions made it ideal for performing asynchronous actions on changes to both documents in Firestore and user accounts in Firebase Authentication.

Moving to Firestore meant formalising the structure of the data that we had hacked together for the prototype to make it more scalable and make sure it plays well with the limits Firestore has around document sizes and nested objects.

To do this, I opted to build a number of sub-collections nested at the appropriate level for the data we stored in the prototype and to use an array of objects that had just enough data to be useful for rendering lists, but contained the ID of a document to link to for the full document.

reciprocal.dev screens for feature lists and selected feature — To render the left screen I denormalised my data model to provide the features list with the ‘just enough’ data (name & description) to render the list in one document. The user would then open the feature and see the right screen which fetches the actual document for that feature to render more information about it

With my data now split into different collections of documents, I needed the means to keep those ‘just enough data’ attributes up-to-date when I performed operations on the full document.

This required creating a means of querying a collection using certain attributes, and I thought these would be useful for others, so here are a few queries I found useful.

Querying a collection for an item

As the data model that I created for reciprocal.dev contains a small subset of data for use in listings, I needed a means of identifying which of these listings I need to update if I remove an item contained in the list.

I found the collectionGroup functionality useful for performing this type of query as it allows for a query to be conducted across a range of collections all with the same keys, so if you have a nested collection such as:

collection1/doc1/collection2
collection1/doc2/collection2

Then you can pass it the collection2 key, and it will search across the collection2 collections under doc1 and doc2.

Finding an item in an array

The majority of my denormalised data uses an array to have the parent record contain a subset of the data stored in documents in a sub-collection.

I have two different types of arrays for this purpose; a list of IDs and a list of objects. The list of IDs is pretty straightforward as you can use the array-contains operation string in a where query against the collection group.

1/**
2 * Given the following document schema at `orgs/1/projects`
3 * 
4 * {
5 *   id: string,
6 *   name: string,
7 *   users: string[],
8 * }
9 */
10
11async function getProjectWithUser(userId: string, db: FirebaseFirestore.Firestore): Promise<object[]> {
12    const querySnapshot = await db.collectionGroup('projects').where('users', 'array-contains', userId).get()
13    const projects = []
14    querySnapshot.forEach((doc: FirebaseFirestore.DocumentData) => {
15      projects.push(doc.data())
16    })
17    return projects
18  }

Using the array-contains operation to query for all documents in projects sub-collection that contain the supplied userId

For an array of objects, you can’t rely on the array-contains approach directly, so I ended up building an array on my documents, especially for performing this type of query. This meant that I could have the object array for using in my app and the array of IDs for querying for documents I need to amend if I delete an item.

Finding an item used as a map key

My data model doesn’t just utilise data from other documents in arrays; it also uses maps for storing information like configuration options and their values across multiple versions.

Luckily, unlike querying arrays, you can query nested maps (up to a depth of 20, I believe), but you can only perform queries checking the map’s value for a key or if the map has a key in it.

1/**
2 * Given the following document schema at `orgs/1/projects`
3 * 
4 * {
5 *   id: string,
6 *   name: string,
7 *   users: Map<string, string>
8 * }
9 * 
10 * Where users is a map of user ID and user role
11 */
12   
13async function getProjectWithUser(userId: string, db: FirebaseFirestore.Firestore): Promise<object[]> {
14  const querySnapshot = await db.collectionGroup('projects').where(`users.${userId}`, '>', "''").get()
15  const projects = []
16  querySnapshot.forEach((doc: FirebaseFirestore.DocumentData) => {
17    projects.push(doc.data()
18  })
19  return projects
20}
21 
22async function getProjectWithUserInRole(userId: string, role: string, db: FirebaseFirestore.Firestore): Promise<object[]> {
23  const querySnapshot = await db.collectionGroup('projects').where(`users.${userId}`, '==', role).get()
24  const projects = []
25  querySnapshot.forEach((doc: FirebaseFirestore.DocumentData) => {
26    projects.push(doc.data()
27  })
28  return projects
29}

Querying for a map key is similar to the array, but to check if a map has a key you need to check the value for the value for the map key sorts after the empty string. For querying the value of the map key, it’s a simple equality check

If you find that you need to query objects at a depth of three or more, you might find it easier to break the document up into sub-collections as you’ll be able to query for that query directly, and this gives your data model more flexibility as you’re not dependent on such a strict structure.

Deleting sub-collections

When you delete a document that has sub-collections in Firestore, it will not delete the document’s sub-collections (or their sub-collections), which means that if not handled, you could end up with a very messy Firestore instance.

While there are scripts that will recursively delete sub-collections, I decided to utilise the Firestore triggers available in Firebase so when a document is deleted, it would delete its immediate sub-collections and have a trigger for each level of the hierarchy. The Firebase documentation contains a set of queries you can use for deleting sub-collections.

This approach meant that deleting large collections isn’t likely to hit the execution time limit of Firebase Functions (9 minutes), and I also have an easier means of retrying a deletion script.

My justification for this is that ultimately once the initial document the user removed has been deleted, there’s no means to perform the deletion of the child sub-collections synchronously, as the user will never request them.

Using triggers to keep the database in shape

As mentioned above, I use the Firestore triggers in a set of Firebase Functions to keep my documents and collections in shape.

There are a number of trigger points you can use within a Firebase Function, but the ones I settled on were:

auth.user().onCreate() — Triggered when a new user is created in the authentication side of Firebase, useful for creating an initial set of data for the user when they sign up
firestore.document().onCreate() — Triggered when a document is created in Firestore, useful for adding a newly created document to its parent or linked documents
firestore.document().onDelete() — Triggered when a document is deleted in Firestore, useful for removing sub-collections when a document is deleted and removing subsets of data for a document that are in other documents across the database.

There is also a trigger for when a document is updated (onUpdate), but I’ve not needed to use this as I’m not storing data that can be changed in other documents just yet.

1/**
2 * Given the following document schema at `orgs/1/projects`
3 * 
4 * {
5 *   id: string,
6 *   name: string,
7 *   users: string[],
8 * }
9 */
10
11import * as functions from 'firebase-functions'
12import * as admin from 'firebase-admin'
13
14admin.initializeApp()
15const db = admin.firestore()
16
17type Project = {
18  id: string,
19  name: string,
20  users: string[]
21}
22
23type ProjectQueryResult = {
24  ref: FirebaseFirestore.DocumentReference<Project>,
25  data: Project
26}
27
28async function getProjectWithUser(userId: string, db: FirebaseFirestore.Firestore): Promise<ProjectQueryResult[]> {
29  const querySnapshot = await db.collectionGroup('projects').where('users', 'array-contains', userId).get()
30  const projects = []
31  querySnapshot.forEach((doc: FirebaseFirestore.DocumentData) => {
32    projects.push({
33      ref: doc.ref,
34      data: doc.data()
35    })
36  })
37  return projects
38}
39
40async function removeUserFromProjects(userId: string, db: FirebareFirestore.Firestore): Promise<void> {
41  const projectsToUpdate = await getProjectsWithUser(userId, db)
42  if (projectsToUpdate.length > 0) {
43    const batch = db.batch()
44    projectsToUpdate.map(({ ref, data }) => {
45      const newUsers = data.users.filter(x => x !== userId)
46      batch.set(ref, {...data, users: newUsers})
47    })
48    await batch.commit()
49  }
50}
51
52exports.removeUserFromProjects = functions.firestore.document('users/{userId}')
53    .onDelete(async (snapshot: functions.firestore.QueryDocumentSnapshot) => {
54      const { id: userId } = snapshot.data() // Can also use `const { userId } = context.params` to extract userId from document path
55      await removeUserFromProjects(userId, db)
56    })

A trigger for removing a user’s ID from all project documents when the user document is deleted

Testing

The Firebase team have made it really easy to run tests thanks to the suite of emulators you can run locally.

All you need to do is ensure that FIRESTORE_EMULATOR_HOST is set in your environment variables to point at the emulator before a test run, and the Firebase library will point its queries at the local emulator.

This makes it really easy to create a suite of integration tests to ensure that your queries work as intended. This can also be used to test your security rules.

I’m using Github Actions for my CI, and I found w9jds/firebase-action really easy to use as it makes sure the emulators are installed on the Github Actions runner for you.

Summary

Working with a NoSQL database requires a little more work than a relational database due to the denormalised approach needed to keep reads quick, but with the right queries and tools, this maintenance doesn’t need to be a chore.

By understanding how Firestore and Firebase work, you can write simple functions to keep everything up-to-date in a scalable manner. Hopefully, the queries I’ve written about will help you achieve this.

Firebase is proving to be a really great set of services for me, and I’m hoping that it will continue to do so in the future. As it’s a Google product, I do worry that it’ll go the way of so many of its other services, but as it’s a developer offering instead of a consumer one, I’m hopeful it will stick around for some time.