The demo scrapes the content of the IPCC website and stores the data in Mediawiki and Wikibase.

This project depends on GNU wget and CPS Wikibase.

CPS Wikibase uses Wikibase docker deployment.

Climate Knowledge Graph tools have been written to make the IPCC report cycle 6 accessible by storing it in Mediawiki and Wikibase.

A Wikibase instance must first be created which will contain both the wikidata and mediawiki content.

Once the site is online and configured custom scripts are utilised to scrape the IPCC site and convert the content to mediawiki markup.

Finally, using an XML configuration file Wikibase is populated using keyword tags to allow SPARQL queries to query for related mediawiki text.

Follow the guide for Wikibase docker deployment but add these extra settings for Mediawiki in LocalSettings.php

$wgEnableUploads = true; # Enable uploads

// Allow images up to 100MB
$wgMaxUploadSize = 104857600;

// Allow page content/wikitext up to 8MB (unit is KiB)
$wgMaxArticleSize = 8192;

$wgShowExceptionDetails = true;

$wgGenerateThumbnailOnParse = false;
$wgThumbnailScriptPath = "{$wgScriptPath}/thumb.php";

These settings are to allow large single page uploads and to stop the automatic generation of thumbnails when importing images which causes a problem. Instead thumbnails are generated as required.

This process for debian downloads and sets up the scrape and import scripts.

sudo apt install python3 python3-venv pandoc python3-magic mercurial
hg clone https://hg.kewl.org/pub/ckg_s2mw
cd ckg_s2mw

python3 -m venv ~/.venvs/ckg_s2mw

or

make venv

TCSH

source ~/.venvs/ckg_s2mw/bin/activate.csh

BASH

source ~/.venvs/ckg_s2mw/bin/activate

make install

hg pull -u
make

A script using wget is provided with a list of URLs to scrape into a local directory.

For this example the directory is accessible via the web but this isn't necessary.

NB a web scrape is a snapshot in time. There is no guarantee the this process is repeatable after the date that these scripts were written to work.

mkdir -p /var/www/htdocs/www.example.com/IPCC
cp etc/wget.sh /var/www/htdocs/www.example.com/IPCC
cp etc/urls*txt /var/www/htdocs/www.example.com/IPCC
cd /var/www/htdocs/www.example.com/IPCC

./wget.sh

After the scrape is complete return to the project directory and setup the environment configuration.

This configuration is used by the python scripts to gain access to the mediawiki and wikibase site deployment.

cp dotenv .env
vi .env

The config requires setting up Mediawiki bot from the wiki at “Special pages → Account management → Bot passwords”.

MEDIAWIKI_URL="https://www.example.com/wiki/"
MEDIAWIKI_HOST="www.example.com"
MEDIAWIKI_USERNAME="User"
MEDIAWIKI_PASSWORD="mybot@xxx"
GATSBY_URL="https://www.ipcc.ch/report/ar6/"
GATSBY_ROOT="/var/www/htdocs/www.example.com/IPCC/Gatsby/www.ipcc.ch/report/ar6/"
WORDPRESS_URL="https://www.ipcc.ch/"
WORDPRESS_ROOT="/var/www/htdocs/www.example.com/IPCC/WordPress/www.ipcc.ch/"
ASSETS_URL="https://www.ipcc.ch/site/"
CACHE_DIR="/var/www/htdocs/www.example.com/IPCC/assets/"

Two scripts are provided to parse the Gatsby and WordPress CMS files to produce intermediary wikitext with Pandoc.

These processes contain various hacks the resultant output is far from perfect.

gatsby -l "etc/urls_gatsby.txt"

wordpress -l "etc/urls_wordpress.txt"

Both processes are IPCC site specific with various workarounds fixing URLs and styling.

The pandoc Wikitext needs to processed and cleaned and this has two stages.

These processes can only be applied once since they modify the pandoc output files from above.

The first parse of the wikitext obtains the media files and uploads them to Mediawiki

wikitext -u

The second process fixes any side effects of the Pandoc conversion process and rewrites the media file links and writes the results to disk.

wikitext -w

This final wikitext process takes the cleaned up wikitext files from above and publishes them on mediawiki

mediawiki -l "etc/urls_gatsby.txt"

mediawiki -l "etc/urls_wordpress.txt"

Using the XML configuration for the IPCC scrape wikibase entries are created.

wbset etc/work.xml

Finally, the main mediawiki page “Main_Page” is written with links to the imported documents.

work