README

###################################
# MINION - scraper daemon framework
#   c[]             ---skrp of MKRX
# scraping work: over-time & many iterations & chronic
# https://ibin.co/38j1IoadTmGz.jpg

# MINIONS ########################
These are examples of diverse scraping methods I have employed.
I treat each target individually to maximize efficency/delays for the servers & the scraper.

# DEMON KEYS #####################
dump     # dump dir
DEBUG    # stderr
LOG      # stdout
PID      # status
BAG      # 24hr workloads
QUE      # current workload
PROG     # current progress
CLEAN    # start automatic
SHUT     # call exit
PAUSE    # call sleep
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
/home/spawn/
/home/host/
/home/sonar/
/home/hive/arki_283/FACE (useragent, protocol, age, name,rep)
/home/hive/arki_283/PID
/home/hive/arki_283/BUG
/home/hive/arki_283/LOG
/home/hive/arki_283/QUE
/home/hive/arki_283/SLEEP
/home/hive/arki_283/SUICIDE
/home/hive/arki_283/POST
/home/hive/arki_283/WORD
/home/hive/arki_283/CODE(popkrip)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
FACE: age, name, rep
PROTO: useragent (handles socket responses)
BUG: debug
LOG: output
QUE: work
POST: fifo out
WORD: fifo in
CODE: popkrip
SLEEP: sleep cmd
SUICIDE: kill cmd
***********************************
archive.org
arxiv.org
europeana.eu
manualslib.com