The scanning has three stages. Initially, the search engine evaluates the possibility of indexing for this reason it is better to develop websites in a way that the search robot has the ability to scan content, considering the site structure. Next, we estimate the possibility of rendering and analyzing the received information. The robot must receive all the necessary content, o complete the operation. Scanning time is the time allocated by the search engine to analyze the site. Time is strictly limited and only the content that is processed by the search engine is included in the index.
Solving SPA indexing problems with free software
Let’s consider solutions of the problem using free software. The client doesn’t need to pay for expensive licenses and develop a solution from scratch. Our task is to solve SPA problems without additional expenses for programmers and software. All you need is to connect and configure a ready-made solution.
The first appropriate solution is called Rendora - this is a new FOSS (free and open-source software) project written in the GO language. Rendora is a dynamic renderer that works with Google headless. In fact, Rendora makes a ready-made HTML server-side render and renders the search engine, and the user works with a standard SPA site. The client receives a standard isomorphic render. There is no need to find a new solution. You just need to connect and configure Rendora.
Rendora functions as a reverse HTTP proxy before the main backend server. The preferred server can work on any technology. For our projects we use x4.cms written in php either Django or Node.js.
The Rendora server checks the requests and in case the required page is requested by the bot, the program sends a command to Google headless to request the search page and then returns the SSR version to the client. In such a way Googlebot gets static HTML. If the page is requested by regular users, Rendora sends the HTML to the client in its original form, without changing anything, like a standard proxy server.
From the visible advantages of the solution, rendora is developed on Golang, which significantly speeds up the work in comparison with similar solutions on Node.js. Caching is implemented for storing SSR pages and subsequent instant issue. You can set up ignoring some unimportant content for indexing, such as types and images, which speeds up Google headless work.
You can read more about configuring and running Rendora on the official github project page.
In the next article we will look at alternative solutions to the problem of rendering and indexing SPA applications.