How to index Firestore Data into Algolia using Cloud Functions.

2019 Update: I’ve written a new post on this topic explaining a simpler process to set up Algolia and Firestore. Check it out!

For the past few days I’ve been struggling a lot to get full-text search working in an app I’m working on. Firestore is great, but they don’t support full-text search. Instead, they recommend a service called Algolia.

While I’m not a programmer (I had to learn to build my business), I usually found my ways around the internet (thank you all!) without much trouble. But this time, it was really really difficult to get this working. Algolia’s documentation is great, but they are not streamlined into one single step-by-step guide specifically for Firestore (which is quite new). Firestore’s documentation is algo usually great, it just happens that for full-text search it is not.

So, hoping it will help someone in the future and hoping this is not crazy wrong, here’s how I did it (with all the references I used).

Step 1: Set up and initialize Firebase SDK for Cloud Functions

Reference: Get Started - Write and Deploy Your First Functions

From the docs:

First, install the Firebase CLI as described in the Firebase CLI Reference. The Firebase CLI requires Node.js and npm, which you can install by following the instructions on https://nodejs.org/. Installing Node.js also installs npm.

Cloud Functions runs Node v.6.11.5, so we recommend that you develop locally with this version.

I used this neat node version controller called n to install the specific node.js version.

Note: The doc says v.6.11.5, but the warnings I got when running told me I should be running v.6.11.1. I ended up using the latter.

So, all the commands so far in a nutshell:

$ npm install -g n
$ n 6.11.1
$ npm install -g firebase-tools
$ firebase login

In case you are using several Firebase projects for different environments (Development, Staging, Production etc.), create aliases to switch between them:

$ firebase use --add

You will be asked which Firebase project you want to create an alias for, and how you’d like to name it.

Now go to the root directory of your project, and run:

$ firebase init

Firebase init will take you through a few questions to help you set up the base files.

Which Firebase CLI features do you want to setup for this folder? Press Space to select features, then Enter to confirm your choices. Functions: Configure and deploy Cloud Functions

What language would you like to use to write Cloud Functions? JavaScript

Do you want to use ESLint to catch probable bugs and enforce style? No

Do you want to install dependencies with npm now? Yes

Remember to include the folder node_modules (that was just created inside the functions folder) in your .gitignore. You don’t want to be commiting that.

That’s it. Your Firebase Cloud Functions CLI is all set up. Next, Algolia.

Step 2: Install Algolia module in the functions folder

I found the way to do that in this YouTube video from Angular, that was referenced in the bottom of an Algolia documentation. Also, in Algolia’s documentation about how to Search on top of your Firebase data with Algolia.

$ cd functions
$ npm install algoliasearch --save
$ npm install dotenv --save
$ npm install firebase --save

This will install the algoliasearch module.

Step 3: Set-up the indexing environment that will not be deployed to Cloud Functions.

What we need to do now is to index all our current data into Algolia. This will be done only once, as in the future you will be creating cloud functions to index automatically all changes in your Firestore database.

For now, however, we need to be able to get all our data indexed and searchable. To do that, Algolia recommends creating an indexing environment.

Create a file called .env inside your function folder. Substitute your values for the placeholders:

ALGOLIA_APP_ID=<algolia-app-id>
ALGOLIA_API_KEY=<algolia-api-key>
ALGOLIA_INDEX_NAME='yourIndexName'
FIREBASE_DATABASE_URL=https://<my-firebase-database>.firebaseio.com

The Firebase Database URL can be found in your Realtime Database console. I know, it should appear in the Firestore console as well, but it doesn’t. The algolia keys can be found in your Algolia Dashboard.

Just be careful: the API key you will use here is the Admin key, which has super powers. Do not place this in any repository or other unsafe places. Keep it safe!

Also, please note that yourIndexName should be the same as your collection name in Firestore (for the script I wrote).

Step 3: Create a new indexing.js file

Inside the functions folder, create a new file called indexing.js. It will be used only once to send the data of each collection in your Firestore database to Algolia.

This is where it got very tricky to me. All the docs were written for Realtime Database, so I had to improvise a little.

In this indexing.js file I wrote this:

// brings all the modules we need
const algoliasearch = require('algoliasearch')
const dotenv = require('dotenv')
const firebase = require('firebase');
const firestore = require('firebase/firestore');
// load values from the .env file in this directory into process.env
dotenv.load();
// initializes the firebase database.
firebase.initializeApp({
  projectId: process.env.FIREBASE_PROJECT_ID,
  databaseURL: process.env.FIREBASE_DATABASE_URL
})
const db = firebase.firestore();
// configure algolia
const algolia = algoliasearch(
  process.env.ALGOLIA_APP_ID,
  process.env.ALGOLIA_API_KEY
);
const index = algolia.initIndex(process.env.ALGOLIA_INDEX_NAME);

What we are doing here is to load all the modules needed, initialize your Firestore database and initialize your Algolia index. I’m using one Algolia index for each Collection in my Firestore database, but please take some time to figure out how you will index yours.

Now, we insert the code that reads all the documents in your collection and sends to Algolia:

var docRef = db.collection(process.env.ALGOLIA_INDEX_NAME);
const records = [];
db.collection(process.env.ALGOLIA_INDEX_NAME).get()
    .then((snapshot) => {
        snapshot.forEach((doc) => {
            // get the key and data from the snapshot
            const childKey = doc.id;
            const childData = doc.data();
            // We set the Algolia objectID as the Firebase .key
            childData.objectID = childKey;
            // Add object for indexing
            records.push(childData);
            console.log(doc.id, '=>', doc.data());
        });
        // Add or update new objects
        index.saveObjects(records).then(() => {
            console.log('Documents imported into Algolia');
            process.exit(0);
        })
        .catch(error => {
            console.error('Error when importing documents into Algolia', error);
            process.exit(1);
        });
    })
    .catch((err) => {
        console.error('Error getting documents', error);
    });

File is ready to run.

The only thing we need to do, temporarily, is to remove the security rules from the Firestore database so unauthenticated clients can read. In the Firebase console, go to the Firestore rules and change to:

service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read, write;
    }
  }
}

Do not forget to set the security rules back on when finishing indexing.

Step 4: Run the script for each collection.

Before running indexing.js, make sure you created the index in Algolia’s dashboard beforehand.

Step 4.1: Change the ALGOLIA_INDEX_NAME value to the name of the Firestore Collection you will index.

Step 4.2: Run indexing.js.

Step 5: Set your Firestore security rules back to secure.

Next step: Create cloud functions to update Algolia on each databse write.

This one is covered here.

Final thoughts

I know it may not be the best way to do it, but it worked for me. I wish the Algolia team takes some time updating their documentation specifically for Firestore, in a step-by-step fashion.

Please let me know if you have any suggestions on how to make this better.