Selenium scrape demo.
sudo apt install python3-full python3-pip mercurial chromium build-essential hg clone https://hg.kewl.org/pub/museum cd museum
Virtual environment
python3 -m venv ~/.venvs/museum
or
make venv
Activate virtual environment
TCSH
source ~/.venvs/museum/bin/activate.csh
BASH
source ~/.venvs/museum/bin/activate
Install dotenv
python -m pip install dotenv
NB Adding dotenv as a pyproject dependency in the virtual environment does not work.
Initial installation of scripts and dependencies
make install
Update installed scripts
make museum
Unique art works.
Example index page
sget https://www.deckenmalerei.eu/malerei-maler_in -b XPATH -v "//div[@class='entityListsPage']" --scroll malerei-maler_in.html
Fetch UUIDs of all art works.
index uuids.csv
URL = https://www.deckenmalerei.eu/malerei-maler_in UUIDS = 2764 TOTAL = 2764 URL = https://www.deckenmalerei.eu/malerei-auftraggeber_in UUIDS = 3625 TOTAL = 4053 URL = https://www.deckenmalerei.eu/malerei-vorlagengeber_in UUIDS = 593 TOTAL = 4121 URL = https://www.deckenmalerei.eu/malerei-technik UUIDS = 3069 TOTAL = 4407 URL = https://www.deckenmalerei.eu/malerei-material UUIDS = 2646 TOTAL = 4549 FILE = uuids.csv
Determine all the art works properties.
analyse uuids.csv properties.txt
de2en.py properties.txt properties.csv
"DE","EN" "ALTERNATIVE NAMEN","ALTERNATIVE NAMES" "AUSRICHTUNG","ALIGNMENT" "BREITE (M)","WIDTH (M)" "DATIERUNG [MALEREI]","DATING [PAINTING]" "DURCHMESSER (M)","DIAMETER (M)" "ERSTELLUNG DES DATENSATZES","CREATION OF THE DATASET" "HAT ARCHITEKT","HAS ARCHITECT" "HAT AUFTRAGGEBER","HAS CLIENT" "HAT BAUER / LANDWIRT","HAS FARMER / FARMER" "HAT BEWOHNER","HAS RESIDENTS" "HAT BEZUGSPERSON","HAS CAREGIVER" "HAT BILDHAUER","HAS SCULPTOR" "HAT BILDSCHNITZER","HAS SCULPTOR" "HAT ENTWERFER","HAS DESIGNER" "HAT KÜNSTLER","HAS ARTIST" "HAT MALER","HAS PAINTER" "HAT MALER -> QUADRATURMALER","HAS PAINTER -> SQUARE PAINTER" "HAT MARMORIERER","HAS MARBLER" "HAT STIFTER","HAS FOUNDER" "HAT STUCKATEUR","HAS PLASTERER" "HAT TEIL","HAS PART" "HAT TISCHLER / KUNSTTISCHLER","HAS JOINER /ART CARPENTER" "HAT VORLAGENGEBER","HAS TEMPLATE CONTRIBUTOR" "HÖHE (M)","ALTITUDE (M)" "ICONCLASS","ICONCLASS" "ID","ID" "INSCHRIFT","INSCRIPTION" "IST DOKUMENTIERT IN","IS DOCUMENTED IN" "IST TEIL VON","IS PART OF" "LETZTE BEARBEITUNG","LAST UPDATE" "LÄNGE (M)","LENGTH (M)" "MATERIAL","MATERIAL" "NAME","NAME" "OBJEKT IM BILDINDEX","OBJECT IN IMAGE INDEX" "OBJEKT IN DER GND","PROPERTY IN THE GND" "POSITION","POSITION" "PRIMÄRE IKONOGRAPHIE","PRIMARY ICONOGRAPHY" "TECHNIK","TECHNOLOGY" "TYP","TYPE" "ZUGEHÖRIGKEIT","BELONGING" "ZUSTAND","CONDITION"
A single art work
sget https://www.deckenmalerei.eu/f65cad80-c7f3-11e9-99f3-c9e55f39fadd -b XPATH -v "//div[@class='dataPage']" --scroll "the-holy-trinity-awaiting-mary-in-heaven.txt"
https://www.deckenmalerei.eu/f65cad80-c7f3-11e9-99f3-c9e55f39fadd