20 Jul 2023
Comments (0)
695 reads

Revolutionize Your Web App: Reduce API Traffic with localStorage and Cookies

For the last couple of years, I have been working on some web app projects that all have the same problem: Some REST API calls are being called repeatedly, but the data doesn't change between the calls. Fetch requests would be sent from the web app both when the user refreshes the page and also in the background using a timeout, and the data that comes back would be exactly the same as the one saved in the application. Changing the setup to use sockets would require lots of work, and therefore instead we had to add fetch requests both when the page loads and every x minutes.

The downside to this approach is that many unnecessary calls to the REST API would be made, and the data coming back would be exactly the same as the one in the web app (Most of the time). One good example is the time-logging web app I built for the company I currently work for. To make administration easier we combined the web app with a 3rd party tool we are using to administrate our kanban boards. This would allow us to easily generate reports to send to our clients by the end of the month, together with the invoice. The 3rd party system is sending a combination of states, deadlines, projects, sub-systems, and versions that are used to customize the data in the reports, and while the datasets are not large, it is being requested many times during the day.

99% of the time these calls are made, the data coming back is exactly the same as the one already saved in the web app. This in my opinion makes these calls unnecessary, and that is why I came up with this solution to reduce the number of API calls made to the server.

Why reduce API traffic?

As of 2023, the internet is the single biggest polluting machine in the world. It accounts for 4% of the world's greenhouse gas emissions and is an ever bigger polluter than the whole aviation sector which only accounts for 3.5%. If the internet was a country it would be the 4th biggest polluter in the world, only exceeded by the US, China, and India.

But why do we never hear about how much the internet pollutes when it is so bad? I think one of the reasons is that most people still don't know that their data pollutes. People think that the world of digital has no pollution involved, that when we upload a picture to Instagram we don't cause pollution, or when we watch a 4K video on YouTube. We tend to think that digitizing papers, pictures, etc. is an environmentally friendly solution, and in many cases, it is correct compared to the alternative. But when we digitize everything other problems arise; the Production of devices, and keeping them running.

Gerry McGovern mentions in his book World Wide Waste that it takes 50 times more minerals to produce an e-reader compared to a regular book, and 40 times more water. Essentially a device would have polluted much more during its production than it would have done during its lifetime, and just to clarify these facts he also gives a comparison. The average American person reads 4 books per year, so to make an e-reader the environmentally friendly option you would have to keep it for 25 years. With the life expectancy of regular electronic devices today you would have to read 100 books per year to make it the better option.

But why and how do electronics pollute? One way that electronics pollute is when it is being produced, and another is to keep them running. We already briefly covered the production, so let's go through what it takes to keep it running.

Essentially your device would be nearly useless if it didn't have an internet connection. The connection to the world is what makes the device so useful. You can contact your family, work on documents with other people, administrate your life, run your business, and all the other things via the internet. But all of these things require an internet connection. This connection is in its simplest form a connection to a spider web stretching worldwide. This web goes everywhere and it has branches that reach into data centers, and each of these data centers hosts websites that provide the data you consume and create. So when you check your feed on Facebook, you will download data from a data center somewhere, and when you upload data such as pictures you send data to be stored in that data center.

Keeping these data centers up and running requires energy, and the servers that are being used are not on the energy-friendly list. Each server has to be powered on 24 hours a day to serve you the data, and in many countries worldwide, energy still comes from burning fossil fuels. If they were turned off, none of the services you like would function. These servers essentially store all the information and take care of all the interactions between you and the internet, and that process requires significant amounts of energy.

So why do we want to reduce the number of API calls? Each server in a data center is being utilized to its most optimal energy consumption level. If this is 75% of its maxed capacity, then data centers will try to keep it there. This is to make sure all machines are doing as much as they can with as little electricity as possible, and the more API calls that are sent the more these servers will have to work. To break down into details; Your device is requesting data from the data center, and the server gets the data from the database, process it, and then sends it to you. If we try to reduce the number of API calls then data centers essentially need fewer servers. Each API call increases power consumption, so reducing it does the opposite.

What is localStorage?

localStorage is a web browser feature that allows websites to store data on a user's computer or device. It provides a way to save key-value pairs locally in the browser and retrieve them later, even if the user closes the browser or restarts their device. localStorage is a part of the Web Storage API, which also includes sessionStorage for storing data that is specific to a particular browsing session.

Here are some key points about localStorage:

Data Storage: localStorage provides a simple and persistent storage mechanism for web applications. It allows websites to store data directly on the user's device, typically in the form of strings.
Key-Value Pairs: Data in localStorage is stored as key-value pairs, similar to an associative array. Each value is associated with a unique key, which can be used to retrieve the data later.
Scope: The data stored in localStorage is scoped to the domain that sets it. This means that each website or domain has its own separate localStorage, and one website cannot access the localStorage of another website.
Size Limit: The amount of data that can be stored in localStorage varies across different web browsers but is generally larger than cookies. The current commonly supported limit is around 5MB.
Data Persistence: Unlike session-based storage like cookies, the data stored in localStorage persists even after the user closes the browser or restarts their device. This makes it suitable for storing information that needs to be accessed across multiple sessions.
No Expiration: By default, the data stored in localStorage does not have an expiration date. It remains available until explicitly removed by the website or cleared by the user.
Security Considerations: It's important to note that while localStorage is a useful tool, it is not designed for storing sensitive or critical data such as passwords or credit card information. Since localStorage data is accessible by JavaScript on the same domain, there is a risk of cross-site scripting (XSS) attacks if malicious code gains access to the data.

Developers commonly use localStorage for various purposes, including:

Storing user preferences or settings.
Caching data to improve performance by reducing server requests.
Saving the user's progress in a web application or game.
Storing tokens or authentication information for persistent login sessions.

When using localStorage, it's essential to handle data securely, validate and sanitize inputs, and be mindful of privacy concerns.

What is a web browser cookie?

A web browser cookie, commonly known as a cookie, is a small piece of data that is stored on a user's computer or device by a website they visit. It is typically used to remember certain information about the user and their preferences, enabling a more personalized browsing experience.

When you visit a website, the server sends a cookie to your browser, which then stores it on your computer or device. The next time you visit the same website, your browser sends the cookie back to the server. This allows the website to recognize you and remember certain information about your previous visits.

Cookies serve various purposes and can be categorized into different types:

Session Cookies: These cookies are temporary and are stored only during your browsing session. They are used to maintain your session information as you navigate through a website. Once you close your browser, session cookies are deleted.
Persistent Cookies: Unlike session cookies, persistent cookies are stored on your computer or device for a longer period. They remain even after you close your browser and can be used to remember your preferences and settings for future visits.
First-party Cookies: These cookies are set by the website you are visiting. They enable the website to remember your preferences and provide a personalized experience.
Third-party Cookies: Third-party cookies are set by domains other than the one you are visiting. They are typically used for advertising and tracking purposes. For example, when you see targeted ads based on your browsing history, third-party cookies are often involved.

LocalStorage with an expiry date

LocalStorage is a great place to store data that doesn't change often, but unfortunately, it does not allow for data expiration which means we cannot specify when the data expires like we can with cookies. This means even after we close the web browser all the data will still exist. The SessionStorage on the other hand has an expiration which invalidates the data once the web session is over, but sometimes we would like to store the data longer than just a single session.

To get around this problem I came up with the solution that we can use cookies to set an expiration date. This will allow us to specifically set an expiration date or time for when certain data should be invalidated, instead of assuming all data needs to expire at the same time. A list of users probably doesn't have to expire at the same time as a list of news.

One thing to remember about Local Storage is that data saved into it will never just be deleted. This is why it is important not to save sensitive data in it since the data could be accessible even to other users on the computer. You can read more about what NOT to store in the local storage further down in the article.

Solution: How to: Combine localStorage with cookies

Now that we have talked about all the boring stuff, it is time to get into how you can implement this in your web application. Normally when requesting data from an API you will create a async fetch request to the server which will then send back data. But for us to be able to first check whether data exist or if it is time to renew it, we need to take another approach.

For those calls you want to implement localStorage, we are going to follow this approach:

First we check whether an expiration cookie exists. If the cookie exists we will check whether the date and time saved in it is older than the time of the request.
If the date has expired we send a request to the API asking for new data. Once the data comes back we renew the data stored in localStorage and update the cookie with a new expiration date.
If the date is not expired we instead load the data already existing in localStorage and use that in our application.

Below you can see the code example that you can copy/paste into your own project. The terms used are:

apiCallStr: This is the string that the fetch function will use to ask information from the API.
setFunction: This is the action that will be used to store the information that comes back from the API. If you are using it in your React application you can insert the set state hook function for example.
cookieName: This is the name of the cookie that will store the expiry date. This name has to be unique because each dataset gets its own expiry date cookie.
localStorageName: This parameter is used to name the local storage data. This name has to be unique because you don't want different datasets to override each other.
expiryTimeInMinutes: This parameter is voluntary which means you don't have to provide it. The function will be using a standard expiry date if this parameter is not provided. But this can be used to describe an expiry date in minutes.

import cookie from 'react-cookies';

async function setLocalStorageData(apiCallStr: string, setFunction, cookieName: string, localStorageName: string, expiryTimeInMinutes?: number): Promise<void> {
    try {
      // FETCH API DATA AND CONVERT TO JSON
      const o = await fetch(`${apiCallStr}`);
      const o_json = o.json();
       
      // RUN THE SET FUNCTION (EXAMPLE: SETSTATE IN REACT TO SAVE DATA TO A STATE)
      setFunction(o_json);
       
      if (o?.status === "Error") {
        // HANDLE ERROR
        return;
      }
       
      // CREATE A NEW EXPIRATION DATE
      const expires: Date = new Date();
       
      // IF EXPIRY DATE IS PROVIDED VIA PROPS
      if (expiryTimeInMinutes) {
        expires.setMinutes(expires.getMinutes() + expiryTimeInMinutes);
      } else {
        expires.setDate(expires.getDate() + 1);
      }
       
      // CREATE EXPIRATION DATE SAVED IN THE COOKIE
      const cookieExp: Date = new Date();
      cookieExp.setDate(cookieExp.getDate() + 2);
      cookie.save(cookieName, expiryTimeInMinutes ? expires.toString() : expires.toDateString(), { expires: cookieExp });
       
      // SAVE API DATA TO LOCAL STORAGE
      localStorage.setItem(localStorageName, JSON.stringify(o));
    } catch (err) {
      // HANDLE ERRORS
    }
  }

  async function useLocalOrApi(expDate, apiCallStr: string, setFunction, cookieName: string, localStorageName: string, expiryTimeInMinutes?: number): Promise<void> {
    try {
      // IF COOKIE WITH EXPIRATION DATE EXISTS
      if (expDate) {
        let parsedExpDate: number = Date.parse(expDate);
        let currentTime: Date = new Date();
         
        if (currentTime.getTime() > parsedExpDate) {
          // IF EXPIRED = RENEW DATA VIA THE API
          setLocalStorageData(apiCallStr, setFunction, cookieName, localStorageName, expiryTimeInMinutes);
        } else {
          // GET THE DATA STORED IN LOCAL STORAGE
          let data: string = await localStorage.getItem(localStorageName);
           
          if (data != null) {
            let dataParsed = await JSON.parse(data);
            // SAVE THE DATA TO THE SET FUNCTION (EXAMPLE = SETSTATE FROM REACT)
            setFunction(dataParsed);
          } else {
            // IF LOCAL STORAGE IS DELETED = FETCH NEW DATA FROM THE API
            setLocalStorageData(apiCallStr, setFunction, cookieName, localStorageName, expiryTimeInMinutes);
          }
        }
      } else {
        setLocalStorageData(apiCallStr, setFunction, cookieName, localStorageName, expiryTimeInMinutes);
      }
    } catch (err) {
      // HANDLE ERRORS
    }
  }
   
  async function fetchData(): Promise<any> {
    try {
      const expDate = await cookie.load('localStorageDataExpDate', { doNotParse: true });
      useLocalOrApi(expDate, `/api-path`, setFunctionToSetState, 'localStorageDataExpDate', 'localStorageDataName');
    } catch (err) {
      // HANDLE ERROR
    }
  }

You can also see a better example here on GitHub.

What not to save in localStorage

While localStorage is a useful tool for storing data in the web browser, there are certain types of data that should never be saved in localStorage due to security and privacy concerns. Here is a list of data that should be avoided:

Passwords: Storing passwords in localStorage is highly discouraged. LocalStorage is not designed to provide secure encryption or protection against unauthorized access. If an attacker gains access to the stored passwords, it could lead to serious security breaches and compromise user accounts.
Credit Card Information: Similarly, sensitive financial data like credit card information should never be stored in localStorage. LocalStorage is accessible by JavaScript running on the same domain, which poses a risk of cross-site scripting (XSS) attacks. Storing credit card information in localStorage increases the likelihood of unauthorized access and potential misuse.
Personally Identifiable Information (PII): PII includes data such as social security numbers, government-issued identification numbers, addresses, phone numbers, or any other information that can be used to identify an individual. Storing such sensitive personal information in localStorage increases the risk of unauthorized access and potential misuse, leading to privacy breaches and identity theft.
Health Information: Any medical or health-related data, such as medical history, prescriptions, or patient records, should not be stored in localStorage. Health information is highly sensitive and subject to strict privacy regulations. Storing such data in localStorage may not comply with legal requirements and could lead to severe privacy violations.
Session Tokens or Authentication Tokens: While storing session tokens or authentication tokens in localStorage can provide persistent login sessions, it is generally considered insecure. LocalStorage is susceptible to cross-site scripting (XSS) attacks, where an attacker could potentially steal these tokens and impersonate the user, leading to unauthorized access to their accounts.
Other Sensitive Data: Any other sensitive data that could compromise user privacy or security should be avoided. This may include financial records, legal documents, confidential business information, or any data that could be used for identity theft, fraud, or unauthorized access.

It's crucial to follow best practices for data security and privacy. For sensitive data, consider server-side storage with proper encryption and access controls. If data storage is necessary for the browser, explore more secure alternatives, such as session-based storage (sessionStorage) or using secure HTTP-only cookies with limited expiration time for managing authentication tokens.

Conclusion

Thank you for reading the article, and I hope you will share it with your community. We need to start thinking of the internet as a polluter and not just the green option to everything.

By now you know how you can reduce the number of API calls in your web applications, and you even have a system you can copy/paste and tweak to your liking. If you find optimizations or ways to make this system even better, please share them with me by emailing me at [email protected] or by commenting below the article.

Author

Michael Andersen

Michael Andersen is the author of Sustainable Web Design In 20 Lessons and the founder of Sustainable WWW (World-wide-web), an organization teaching sustainable practices. With a passion for web design and the environment, Michael solves puzzles to make the internet more sustainable.