How to lazy load section content with the whiteCollar

Deprecation

This guide is only for tenants using PhantomJS.

Newer tenants use Puppeteer, which has a specific feature to extract lazy loaded content.

Some websites have articles that only show when scrolling down or by clicking on a Load more... button.

In this case, the extraction can fail because these articles are loaded by a script and do not appear in the source code.

These are some approaches that can be used to manage this situation:

1. Open Chrome DevTools and, on the Network tab, inspect the requested URL that is triggered when the articles are loaded.

Many WordPress sites use a /page/2/ URL pattern where they show more articles. In this case, it is only needed to add it in definition.json

2. Use Ajax

You can also extract the articles from the requested URL with ajax:

function loadMoreNewsUpdate(callback) {
  $.ajax({
    type: 'GET',
    url: 'https://example.com/?ajax-request=jnews&action=jnews_module_ajax_jnews_block_28&data[attribute][post_type]=post&data[current_page]=2&data[attribute][number_post]=5&data[attribute][include_category]=16',
    success: function(response) {
      var wrapper = document.createElement('div');

      wrapper.className = 'mrf-loadMore__newsUpdate';
      wrapper.innerHTML = response.content;

      document.body.appendChild(wrapper);

      callback && callback();
    },
    error: function() {
      return callback && callback();
    }
  });
}