How Sambal SOS Scaled (The Complete Development Workflow)

A to Z

It’s been about 2 weeks since the inception of Sambal SOS (formerly the Bendera Putih App). A roller-coaster of emotions, lengthy code sprints, bug hunting, over 250 commits, 45 pull requests, 422 reports and 25k users later, I sit down to write this article.

Sambal SOS, in its early stages, wasn’t built to scale. We started it as a hobby/hackathon project; we didn’t expect much traffic, let alone, attention at a national level. When it started to surge, we started to realise what we built and it was surreal. At the same time, issues arose and we had to find quick fixes to keep the build stable. We realised the result of a rapid workflow, bypassing testing and staging and going directly from development to production.

However, it was heartwarming to see real people using it when it finally went out there and we knew we had to scale.

I’m here to go through every step of the way, all the issues we faced and the lessons we learnt in the evolution of Sambal SOS.

Beta Release (Friday- 2 July)

Before this, we had announced on Developer Kaki that the Bendera Putih app was in development and we would be releasing Beta soon. The day had come. We had a stable build (or so we thought).

We didn’t have a backend server yet, so we made the first big mistake: choosing Firebase. The React frontend was connected to Firebase and deployed live…and we were on the free version of Firebase. So when traffic started to climb that night, I couldn’t take my eyes off the console, just waiting for production to crash as we exceeded all the set quotas.

Issue 1: Firebase

Use Firebase for hosting hobby projects, it’s not for anything that’s meant to scale. Luckily, we hadn’t invested too much time in setting it up. Migrating out of firebase and firestore was seamless after the 2 days Sambal SOS was hosted there.

Issue 2: Incorrectly configuring cookies

If you look at our Redux setup, you would see this login, logout flow to handle the authentication state across the application.

import { createSlice } from "@reduxjs/toolkit";
import cookie from "js-cookie";export const authSlice = createSlice({
  name: "auth",
  initialState: {
    user:
      cookie.get("user") !== undefined ? JSON.parse(cookie.get("user")) : {},
    isAuthenticated: cookie.get("accessToken") ? true : false,
    accessToken: cookie.get("accessToken") ? cookie.get("accessToken") : "",
  },
  // fix all this
  reducers: {
    LOGIN: (state, action) => {
      state.user = action.payload.user;
      state.accessToken = action.payload.accessToken;
      state.isAuthenticated = true;
      cookie.set("user", JSON.stringify(action.payload.user));
      cookie.set("isAuthenticated", true);
      cookie.set("accessToken", action.payload.accessToken);
    },
    LOGOUT: (state) => {
      state.isAuthenticated = false;
      state.accessToken = "";
      state.user = null;
      cookie.set("user", null);
      cookie.set("isAuthenticated", false);
      cookie.set("accessToken", "");
    },
    UPDATE: (state, action) => {
      state.user = action.payload;
    },
  },
});export const { LOGIN, LOGOUT, UPDATE_USER } = authSlice.actions;export const selectIsAuthenticated = (state) => state.isAuthenticated;
export const selectUser = (state) => state.user;
export default authSlice.reducer;

However, we had forgotten to set the cookies upon login but chose to read it to gauge whether the user was logged in. Hence, every time you refreshed the page, you were redirected back to log in. Without realising the mistake was in the Redux setup, we started blaming Firebase (well, Firebase was problematic in so many ways, but certainly did not cause issues with the authentication flow).

Then, people started complaining about blank screens when loading the page on Facebook’s internal browser. Properly setting the cookies fixed the “blank page” issue.

Issue 3: Bing Maps

We didn’t have funding, so for the real-time map, we used Bing Maps. It had a generous free quota, which was enough to sustain us for the little time we thought we’d last. When integrated with React, it was laggy, froze most of the time and configuring tooltips became a nightmare. The map was barely navigable and we needed to find a solution for this quick.

Our post on Developer Kaki attracted a few sponsors, one of whom offered to host our infrastructure on his cloud. He also offered to pay for our Google Maps API in the initial stages. The next day, we stripped down the whole page, and migrated to react-google-maps, with newfound support. Changes were profound; we immediately started to notice the performance boost and incredible documentation made configuration all the more flexible.

Vercel and Google Cloud Migration

The app had blown up, people had started to use it. We were seeing flags on the map, and it was time to scale.

Our architecture now involved a React.js frontend (built with Chakra UI), Node.js (Express) API for the backend and PostgreSQL for the database. Here’s where each part of the stack lived:

  1. Frontend -> Vercel
  2. Backend -> GCP App Engine
  3. Database -> Cloud SQL

Now, this was a stack built to scale to the moon. Our sponsor had gotten us a 10GB SSD DB instance and App Engine was blazing fast. While deploying and setting up the database was a hassle (we ended up skipping Sequelize migrations), it was finally up by Sunday morning.

It was time to enter production.

Production (Sunday- 4 July)

The Beta Release was only for the developer community, so this time, we announced it to everyone. Our social media platforms were set up, design assets were built and we released through a Facebook post, that eventually went viral. A Malaysian journalist tweeted about us, and it got over 6k retweets, overnight, the Bendera Putih App blew up. And with this, a multitude of issues arose:

Service Workers

Progressive Web Apps (PWA). Everyone loved them, so we thought that the Bendera Putih App should be a PWA. Users could then download the app onto their home screens and use it as an app. However, to do this right, you have to write your service workers properly. These are essentially little functions that tell the browser how to treat your page as an app.

The problem is: we wrote them wrong. There were faulty service workers, not caching the page properly, so upon the second visit, you would be met with a blank page.

We also had a faulty service worker that didn’t cache because the file service-workers.js was wrongly named as serviceWorkers.js. We didn’t have time to write new ones or experiment with them, so the simplest fix was to de-register all the service workers and remove PWA functionality. So, a new set of service workers were pushed live, with the sole purpose of killing their predecessors.

var CACHE_NAME = "pwa-task-manager";
var urlsToCache = [];

self.addEventListener("install", function (e) {
  self.skipWaiting();
});

self.addEventListener("activate", function (e) {
  self.registration
    .unregister()
    .then(function () {
      return self.clients.matchAll();
    })
    .then(function (clients) {
      clients.forEach((client) => client.navigate(client.url));
    });
});

Printing Incorrect Addresses

While there was a search bar that picked up the exact geolocation, in the UI, it was geocoded to a full address. The problem was geocoding was often very inaccurate. If you reported an incident 100m away from a house in a barren area, it would return the closest registered property.

Within an hour, we removed this option and chose to add a button that redirected to google maps. This would take them to the exact geolocation and not a generic address.

Google Maps Javascript API

The price of the Google Maps Javascript API grew like bitcoin and our sponsor urged us to find proper corporate sponsorship to keep the maps feature alive. We later obtained credits via the Maps Platform for Crisis Responders.

Incorrectly loading Google Maps

When setting up Google Maps on React, you have to dynamically load the google maps script and only load the map after it was fully loaded. This was initially not set up, so upon the first load, you would see an internal server error. We rectified this by using the useJsApiLoader function, which returned a loading state.

The map component was only loaded after everything was fully loaded.

const { isLoaded } = useJsApiLoader({
    id: "google-map-script",
    googleMapsApiKey: process.env.REACT_APP_GOOGLE_MAPS_API_KEY,
    libraries,
  });
{loading || !isLoaded ? (
        <Center
          h="80vh"
          flexDirection="column"
          justifyContent="center"
          alignItems="center"
        >
          <Spinner />
        </Center>
      ) : (
        <GoogleMap
          mapContainerStyle={mapContainerStyle}
          zoom={8}
          center={center}
          onLoad={onLoad}
          onUnmount={onUnmount}
          options={options}
          onClick={() => {
            setModalVisible(false);
            setFoodbankModalVisible(false);
          }}
        >

Cloudinary

We chose Cloudinary when it was still seen as a hobby project. We let users upload full-resolution photos onto Cloudinary from the frontend (Cloudinary really had no reason to properly downscale it, because the faster we met our quota, the sooner we would have to pay for their premium plan).

There were 8MB images sitting on Cloudinary, clogging up space and bandwidth, even at the lowest setting.

Eventually, we started using a sponsored AWS VPS running MinIO. All new images started going there. Image compression using Sharp on the Node server was also very effective, taking 8MB images and scaling them down to 10KB.

let randomFileName = uuidv4() + '.' + req.file.originalname.split('.')[req.file.originalname.split('.').length - 1];
    let filename = req.file.originalname;
    sharp(req.file.buffer)
        .jpeg({
            quality: 30,
        })
        .withMetadata()
        .rotate()
        .toBuffer()
        .then((data) => {
            minioClient.putObject('reports', randomFileName, data, function(err, etag) {
                if(err) {
                    res.status(500).send({
                        message: err,
                        status: 'failed to upload to minio'
                    })
                } else {
                    res.status(200).send({
                        message: etag,
                        status: `${randomFileName} uploaded successfully`,
                        secure_url: minioClient.protocol + '//' + minioClient.host + ':' + minioClient.port + '/reports/' + randomFileName
                    })
                }
            })
        })

Flipping Latitude and Longitude (for about an hour)

Well, this was embarrassing. In one of the functions that fetched the current location, the latitude and longitude were swapped. This resulted in food banks showing up in the Atlantic Ocean. We quickly fixed this but had to manually swap the existing records to correct them.

An Increasingly Cluttered Codebase

When it became clear that the team was growing and more people were actively pushing, it was time to enforce linting and pull request procedures so we could maintain the code quality.

Multi-lingual App

It started out as a single-language app. Volunteers helped translate. It’s currently in English, Malay and Chinese; we’ll add Tamil soon.

We used react-i18n-next. And this was the setup:

import i18n from "i18next";
import { initReactI18next } from "react-i18next";
import en from "./translations/en/translations.json";
import malay from "./translations/malay/translations.json";
export const resources = {
  English: {
    translation: en,
  },
  Malay: {
    translation: malay,
  },
};
i18n
  .use(initReactI18next)
  .init({
    resources,
    lng: "English",
    interpolation: {
      escapeValue: false,
    },
  });
export default i18n;

There was a toggle button that allowed users to switch between the languages.

Something

Foodbank Moderation

The requests to add foodbanks to the platform started to grow and it required modifying their statuses in real-time based on their stock. We started collaborating with r/Malaysia, which had a task force for this, giving us access to rich information about these food banks. We’re currently working on fetching it from their real-time APIs rather than static JSON updated via a Google Sheets workflow.

Main and Production should be different branches

The main branch on GitHub should be the primary development branch and production should be the branch Vercel/the frontend hosting is listening to changes on. For a long time, whenever a PR was merged into main, a redeploy would be triggered.

The engineering team is now bigger. We have Malaysian developers, some even in other parts of the world, working on issues and new features. This is now a stable build, so it’s time to process all the feature requests and start integrating them.

Sambal SOS is a completely open-source project, with the repository linked here:

GitHub

We’ve learnt so much, but these were a few of the mistakes we made along the way and how we eventually fixed them. The team is incredibly proud of how far the app has come, especially the incredible support we’ve received from the developer community and sponsors. This is only the beginning.

In case you haven’t checked out the app yet, have a look here:

App

Comments

Thanks for reading ❤️

You can comment by replying to the issue for this post.

There’s no comments yet, be the first to leave one!

Other posts