Skip to content

Commit

Permalink
First commit.
Browse files Browse the repository at this point in the history
  • Loading branch information
odkr committed May 9, 2018
0 parents commit 47fe4fc
Show file tree
Hide file tree
Showing 26 changed files with 2,694 additions and 0 deletions.
26 changes: 26 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
test: test-doc test-call-citeproc test-dont-call-citeproc

test-doc:
rm -f test/doc-is.html
pandoc --lua-filter ./pandoc-zotxt.lua -F pandoc-citeproc \
-o test/doc-is.html test/doc.md
cmp test/doc-is.html test/doc-should.html

test-call-citeproc:
rm -f test/call-citeproc-is.html
pandoc --lua-filter ./pandoc-zotxt.lua \
-o test/call-citeproc-is.html test/call-citeproc.md
cmp test/call-citeproc-is.html test/call-citeproc-should.html

test-dont-call-citeproc:
rm -f test/dont-call-citeproc-is.html
pandoc --lua-filter ./pandoc-zotxt.lua \
-o test/dont-call-citeproc-is.html test/dont-call-citeproc.md
cmp test/dont-call-citeproc-is.html test/dont-call-citeproc-should.html

performance-comparison:
time pandoc -F pandoc-zotxt -o /dev/null test/long.md
time pandoc --lua-filter ./pandoc-zotxt.lua -o /dev/null test/long.md

.PHONY: test test-doc test-call-citeproc test-dont-call-citeproc \
performance-comparison
135 changes: 135 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
================
pandoc-zotxt.lua
================

``pandoc-zotxt.lua`` looks up sources of citations in Zotero and adds
their bibliographic data to the metadata of the document, where it
can be read by ``pandoc-citeproc``.

You need the zotxt_ plugin for Zotero. Citations should be inserted
as so-called easy citekeys. See the documentation of *zotxt* for details.

See the `manual page <man/pandoc-zotxt.lua.rst>`_ for more details.



Installing ``pandoc-zotxt.lua``
===============================

You use ``pandoc-zotxt.lua`` **at your own risk**. You have been warned.

You need Pandoc_ 2.0 or later. If you are using an older version of Pandoc,
try `pandoc-zotxt <https://github.com/egh/zotxt>`_,
which works with Pandoc 1.12 or later (but also requires Python_ 2.7).

1. Download the `current release
<https://codeload.github.com/odkr/pandoc-zotxt/tar.gz/v0.1>`_.
2. Unpack it.
3. Copy the whole directory to the ``filters``
subdirectory of your Pandoc data directory.

Where your Pandoc data directory is located depends on your operating system.
``pandoc --version`` will tell you. Consult the Pandoc manual for details.

You may also want to copy the manual page to wherever your system stores manual
pages; typically, this is ``/usr/local/share/man/``.

If you are using a Unix-ish operating system, you can do all of the above by::

PANDOC_DATA_DIR=$(pandoc --version |
sed -n 's/^Default user data directory: //p')
mkdir -p "${PANDOC_DATA_DIR:?}/filters"
cd "${PANDOC_DATA_DIR:?}/filters"
curl https://codeload.github.com/odkr/pandoc-zotxt.lua/tar.gz/v0.1 |
tar -xz
sudo cp pandoc-zotxt.lua-0.1/man/pandoc-zotxt.lua.1 \
/usr/local/share/man/man1


``pandoc-zotxt.lua`` vs ``pandoc-zotxt``
========================================

I started to write ``pandoc-zotxt.lua`` because I had hoped that I could write
a faster replacement for ``pandoc-zotxt``. Unfortunately, Pandoc_ does *not*
support LuaSocket_ (a library for, among other things, retrieving data via a
network) and only provides a blocking method to fetch data from networks
itself. So, there is no way to retrieve data for multiple citation items
concurrently. As a consequence, ``pandoc-zotxt.lua`` is about as fast as
``pandoc-zotxt``.

+------------------------------------+---------------------------------------+
| ``pandoc-zotxt.lua`` | ``pandoc-zotxt`` |
+====================================+=======================================+
| Requires only Pandoc_ 2.0 | Requires Pandoc_ 1.12 and Python_ 2.7 |
+------------------------------------+---------------------------------------+
| Apparently, a tiny bit faster | Apparently, a tiny bit faster |
| for long and/or complex documents. | for short and/or simple documents. |
| (But you won't notice.) | (But you won't notice.) |
+------------------------------------+---------------------------------------+


Test suite
==========

For the test suite to work, you need Zotero_ and the sources that are cited
in the test documents. You can import those sources from the file ``items.rdf``
in the directory ``test``. To run the test suite, just say::

make test


Documentation
=============

See the `manual page <man/pandoc-zotxt.lua.rst>`_
and the source for details.


Contact
=======

If there's something wrong with ``pandoc-zotxt.lua``, `open an issue
<https://github.com/odkr/pandoc-zotxt.lua/issues>`_.


License
=======

Copyright 2018 Odin Kroeger

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


Further Information
===================

GitHub:
<https://github.com/odkr/pandoc-zotxt.lua>


See also
========
<https://github.com/egh/zotxt>


.. _zotxt: https://github.com/egh/zotxt
.. _Zotero: https://www.zotero.org/
.. _Pandoc: https://www.pandoc.org/
.. _Python: https://www.python.org/
.. _LuaSocket: https://github.com/diegonehab/luasocket
8 changes: 8 additions & 0 deletions lib/luarocks/rocks-5.3/lunajson/1.2-0/doc/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# 1.2.0

* For ease of embedded use, the decoder and the SAX parser are made self-contained one file implementation.
* `1` is decoded as an integer on Lua 5.3.
* Number parsing routine is changed. Now `-?[0-9][-+.A-Za-z0-9]*` is detected as a number, and its conformance to JSON spec is checked.
* Automated testing.
* Cool logo :)
* Bug fixes.
21 changes: 21 additions & 0 deletions lib/luarocks/rocks-5.3/lunajson/1.2-0/doc/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The MIT License (MIT)

Copyright (c) 2015-2017 Shunsuke Shimizu (grafi)

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
84 changes: 84 additions & 0 deletions lib/luarocks/rocks-5.3/lunajson/1.2-0/doc/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# ![Lunajson](logo/lunajson.png)
[![CircleCI](https://circleci.com/gh/grafi-tt/lunajson.svg?style=shield)](https://circleci.com/gh/grafi-tt/lunajson)

Lunajson features SAX-style JSON parser and simple JSON decoder/encoder. It is tested on Lua 5.1, Lua 5.2, Lua 5.3, and LuaJIT 2.0.
It is written only in pure Lua and has no dependencies. Even so, decoding speed matches lpeg-based JSON implementations because it is carefully optimized.
The parser and decoder reject input that is not conformant to the JSON specification (ECMA-404), and the encoder always yields conformant output.
The parser and decoder also handle UTF/Unicode surrogate pairs correctly.

## Install
luarocks install lunajson

Or you can download source manually and copy `src/*` into somewhere on your `package.path`.

## Simple Usage
local lunajson = require 'lunajson'
local jsonstr = '{"Hello":["lunajson",1.5]}'
local t = lunajson.decode(jsonstr)
print(t.Hello[2]) -- prints 1.5
print(lunajson.encode(t)) -- prints {"Hello":["lunajson",1.5]}

## API
### lunajson.decode(jsonstr, [pos, [nullv, [arraylen]]])
Decode `jsonstr`. If `pos` is specified, it starts decoding from `pos` until the JSON definition ends, otherwise the entire input is parsed as JSON. `null` inside `jsonstr` will be decoded as the optional sentinel value `nullv` if specified, and discarded otherwise. If `arraylen` is true, the length of an array `ary` will be stored in `ary[0]`. This behavior is useful when empty arrays should not be confused with empty objects.

This function returns the decoded value if `jsonstr` contains valid JSON, otherwise an error will be raised. If `pos` is specified it also returns the position immediately after the end of decoded JSON.

### lunajson.encode(value, [nullv])
Encode `value` into a JSON string and return it. If `nullv` is specified, values equal to `nullv` will be encoded as `null`.

This function encodes a table `t` as a JSON array if a value `t[1]` is present or a number `t[0]` is present. If `t[0]` is present, its value is considered as the length of the array. Then the array may contain `nil` and those will be encoded as `null`. Otherwise, this function scans non `nil` values starting from index 1, up to the first `nil` it finds. When the table `t` is not an array, it is an object and all of its keys must be strings.

### lunajson.newparser(input, saxtbl)
### lunajson.newfileparser(filename, saxtbl)
Create and return a sax-style parser context, which parses `input` or a file specified by `filename`. `input` can be a string to be parsed, or a function that returns the next chunk of a data as a string to be parsed (or `nil` when all data is yielded). An example function for `input` follows (this sample is essentially same as the implementation of `newfileparser`). Note that `input` will never be called once it has returned `nil`.

local fp = io.open("myfavorite.json")
local function input()
local s
if fp then
s = fp:read(8192)
if not s then
fp:close()
fp = nil
end
end
return s
end

`saxtbl` is a table of callbacks. It can have the following functions. Those functions will be called on corresponding events, if it is in the table.

- startobject()
- key(s)
- endobject()
- startarray()
- endarray()
- string(s)
- number(n)
- boolean(b)
- null()

A parser context maintains the current parse position, initially 1.

#### parsercontext.run()
Start parsing from current position. If valid JSON is parsed, the position moves to just after the end of this JSON. Otherwise it errors.

#### parsercontext.tellpos()
Return current position.

#### parsercontext.tryc()
Return the byte of current position as a number. If input is ended, it returns `nil`. It does not change current position.

#### parsercontext.read(n)
Return the `n`-length string starting from current position, and increase the index by `n`. If the input ends, the returned string and the updated position will be truncated.

## Benchmark
Following graphs are the results of the benchmark, decoding [`simple.json`](test/decodeparse/benchjson/simple.json) (about 750KiB) 100 times and encoding [`simple.lua`](test/encode/benchdata/simple.lua) (the decoded result of `simple.json`) 100 times. I conducted benchmarks of lunajson 1.0, [dkjson 2.5](http://dkolf.de/src/dkjson-lua.fsl/home) and [Lua CJSON 2.1.0](http://www.kyne.com.au/~mark/software/lua-cjson.php). Dkjson is a popular JSON encoding/decoding library in Lua, which is written in Lua and optionally uses [lpeg](http://www.inf.puc-rio.br/~roberto/lpeg/) to spped up decoding. Lua CJSON is a JSON encoding/decoding library implemented in C and is inherently fast.

![The graph of decoding benchmark results](result/decode-simple.png)

![The graph of encoding benchmark results](result/encode-simple.png)

This benchmark is conducted in my desktop machine that equips Core i5 3550K and DDR3-1600 memory. Lua implementations and concerning modules are compiled by GCC 4.9.2 with `-O2 -march=ivybridge -mtune=ivybridge` options. The versions of lua implementations are the newest official releases at the time of benchmark. The version of lpeg is 0.12.2.

In this benchmark Lunajson performs well considering that it is implemented only in standard Lua, especially in LuaJIT 2.0 benchmark. Lunajson also supplies incremental parsing in a SAX-style API, therfore you don't have to load whole large JSON files into memory in order to scan the information you're interested in from them. Lunajson is especially useful when non-standard libraries cannot be used easily or incremental parsing is favored.
31 changes: 31 additions & 0 deletions lib/luarocks/rocks-5.3/lunajson/1.2-0/lunajson-1.2-0.rockspec
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package = "lunajson"
version = "1.2-0"
source = {
url = "git://github.com/grafi-tt/lunajson.git",
tag = "1.2"
}
description = {
summary = "A strict and fast JSON parser/decoder/encoder written in pure Lua",
detailed = [[
Lunajson features SAX-style JSON parser and simple JSON decoder/encoder. It is tested on Lua 5.1, Lua 5.2, Lua 5.3, and LuaJIT.
It is written only in pure Lua and has no dependencies. Even though, since it is carefully optimized, decoding speed even matches to other lpeg-based JSON modules.
The parser and decoder reject inputs not conforms the JSON specification (ECMA-404), and the encoder always yields outputs conforming the specification.
The parser and decoder also handle surrogate pair correctly.
]],
homepage = "https://github.com/grafi-tt/lunajson",
maintainer = "Shunsuke Shimizu",
license = "MIT/X11"
}
dependencies = {
"lua >= 5.1"
}
build = {
type = 'builtin',
modules = {
['lunajson'] = 'src/lunajson.lua',

['lunajson.decoder'] = 'src/lunajson/decoder.lua',
['lunajson.encoder'] = 'src/lunajson/encoder.lua',
['lunajson.sax' ] = 'src/lunajson/sax.lua',
}
}
16 changes: 16 additions & 0 deletions lib/luarocks/rocks-5.3/lunajson/1.2-0/rock_manifest
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
rock_manifest = {
doc = {
["CHANGELOG.md"] = "b173b6220589bd7e512cb5ea1f009060",
LICENSE = "2001d13200cac722635f81768f80463e",
["README.md"] = "c6d5d3393e5d54b40abd764d1eb24dcb"
},
lua = {
lunajson = {
["decoder.lua"] = "a02440404a53a1c6d7cd70c9362884f4",
["encoder.lua"] = "197d6edcbc03cea7db3d918493c11616",
["sax.lua"] = "bf58ae0c4a877ba4e49ee31b05a14fed"
},
["lunajson.lua"] = "64bce3b8bcfa53c02e64fd924c1cf4b4"
},
["lunajson-1.2-0.rockspec"] = "e16bb1378fa36d6a662964ea925efa48"
}
49 changes: 49 additions & 0 deletions lib/luarocks/rocks-5.3/manifest
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
commands = {}
dependencies = {
lunajson = {
["1.2-0"] = {
{
constraints = {
{
op = ">=",
version = {
5, 1, string = "5.1"
}
}
},
name = "lua"
}
}
}
}
modules = {
lunajson = {
"lunajson/1.2-0"
},
["lunajson.decoder"] = {
"lunajson/1.2-0"
},
["lunajson.encoder"] = {
"lunajson/1.2-0"
},
["lunajson.sax"] = {
"lunajson/1.2-0"
}
}
repository = {
lunajson = {
["1.2-0"] = {
{
arch = "installed",
commands = {},
dependencies = {},
modules = {
lunajson = "lunajson.lua",
["lunajson.decoder"] = "lunajson/decoder.lua",
["lunajson.encoder"] = "lunajson/encoder.lua",
["lunajson.sax"] = "lunajson/sax.lua"
}
}
}
}
}
Loading

0 comments on commit 47fe4fc

Please sign in to comment.