User:Luigi.scarso/luatex lunatic
Contents
Introduction
!! W A R N I N G !!
!! THIS CODE IS HIGHLY EXPERIMENTAL !!
!! THIS CODE WORKS ONLY UNDER LINUX !!
!! THIS CODE WORKS ONLY WITH luatex 0.40.5 !!
!! THIS CODE WORKS ONLY WITH ConTeXt ver: 2009.06.14 21:01 MKIV
luatex_lunatic is a modification of lua side of luatex to host a python interpreter inside lua (see lunatic-python) .
I have made a set of patches because :
- by design, lua in luatex doesn't permit dynamic loading ("Dynamic loading of .so and .dll files is disabled on all platforms." see "LUA changes" in manual/luatexref-t.pdf of src and/or svn dist.)
- to prevent symbols collisions between luatex and an arbitrary library
A python interpreter hosted in luatex can make easier to use existing python's bindings; also the ctype modules (included in python releases at from 2.5 version) permits a binding to a .so without using SWIG or similar (almost like loadlib of lua) .
As general rule, I want the smallest set of patches ; so, for example, I have choose to not use a system libpng, even if it's easy to modify build.sh. to do so.
Also, I'm not concerned about portability: I will talk only about Linux .
Please note that luatex_lunatic was born to answer to (my) question "Can I apply lunatic to luatex ?" and nothing else.
Every others meanings can be interestring, but are not mine , and I will not partecipate to any discussion about that.
How to
I prefear to put a bash script that can help to compile a luatex-lunatic binary. It's clear that some skills are needed, so I hope in this way to avoid useless questions . Of course, I will try to correct all errors .
The trick is hide all, then unhide what is needed, then make only what is needed and in the end re-compile luatex .
The changes are :
- in source/texk/web2c/luatexdir/lua51/loadlib.c:
69c69 < void *lib = dlopen(path, RTLD_NOW); --- > void *lib = dlopen(path, RTLD_NOW|RTLD_GLOBAL); 74a75 >
- in source/texk/web2c/luatexdir/am/liblua51.am
12c12 < liblua51_a_CPPFLAGS += -DLUA_USE_POSIX --- > liblua51_a_CPPFLAGS += -DLUA_USE_LINUX
- in source/texk/web2c/Makefile.in:
101c101 < @MINGW32_FALSE@am__append_15 = -DLUA_USE_POSIX --- > @MINGW32_FALSE@am__append_15 = -DLUA_USE_LINUX 1575c1575 < $(CXXLINK) $(luatex_OBJECTS) $(luatex_LDADD) $(LIBS) --- > $(CXXLINK) $(luatex_OBJECTS) $(luatex_LDADD) $(LIBS) -Wl,-E -uluaL_openlibs -fvisibility=hidden -fvisibility-inlines-hidden -ldl -lreadline -lhistory -lncurses
So, first compile as usual
$>build.sh
Then use trick.zip" to unhide symbols from liblua51.a and recompile luatex (put it at the same level of build.sh) .
Python packages
These are python packages that are not in standard libraries ; "✔" means that I have made only small lua wrapper .
Only for PythonImagingLibrary (PIL) I have made a example.
- numpy ✔
- scipy ✔
- matplot ✔
- odfpy ✔
- PIL, python imaging library ✔ PIL
For these, I stille have to decide what todo.
- TO FIX ; pygegl (I like its syntax, but need too much and it's easy only in python)
- TODO: binding of libtiff with ctypes (or use gdal ?)
- TODO :gle (for Massimiliano "Max" Dominici, GUIT)
- TODO : using the binding of VIPS
Bindings
These are shared lib that come with native python binding, or eventually I have made a wrapper using ctypes .
- ghostscript 8.64 ✔ here
- graphviz 2.22.0 ✔
- ImageMagick-6.4.9 with pythonmagickwand ✔
- fontforge 20090224 ✔ here Useful to check symbols collision, and if one want to play with the last fontforge, eg to draw the outline of a glyph .
- R-2.8.1 with rpy2-2.0.3 (For Maurizio "Mau" Himmelman , GUIT) ✔ here (see also here) .
- quantlib 0.9.7 ✔ need an example with output in pdf
- dbxml-2.4.16 ✔
Dedicated systems
When we have a huge and coordinate set of *so with python binding, it' better to dedicate a specific luatex-lunatic . As example I choose sagemath and ROOT .
SageMath
From sagemath :
Sage is a free open-source mathematics software system licensed under the GPL. It combines the power of many existing open-source packages into a common Python-based interface.
Given that sagemath
is rooted on a CPtyhon , we can try to use sagemath as "plugin" for luatex
instead of use latex as external process of sagemath.
sagemath comes with own python, and we must use it; in this case I put all stuffs under a separated lunatic-python-SAGEMATH .
HOME_LUN=/opt/luatex/luatex-lunatic cd $HOME_LUN cp -r lunatic-python lunatic-python-SAGEMATH cd lunatic-python-SAGEMATH ## ## We prepare new bridge ## rm -r build $HOME_LUN/sage/local/bin/python setup.py build cd $HOME_LUN mkdir tests-SAGEMATH ## ## I have already installed prev. python.so, I don't want mess things ## ln -s ../lunatic-python-SAGEMATH/build/lib.linux-i686-2.5/python.so ln -s ../luatex/build/texk/web2c/luatex ## ## now we need to setup sagemath ## cd $HOME_LUN/sage/local/bin . sage-env cd $HOME_LUN
We need also a first run of sagemath:
cd sage ./sage ## ## now exit from sage ##
That is .
Now we need a stub, because usually sage is used as an interactive shell:
## sagestub.py from sage.all_cmdline import *
OK let's start with an example with mixed code (or, better, 'messed code'):
\startluacode function test_ode(graphout) require("python") pg = python.globals() SAGESTUB = python.import("sagestub") sage = SAGESTUB.sage python.execute([[ def f_1(t,y,params): return[y[1],-y[0]-params[0]*y[1]*(y[0]**2-1)] ]]) python.execute([[ def j_1(t,y,params): return [ [0,1.0],[-2.0*params[0]*y[0]*y[1]-1.0,-params[0]*(y[0]*y[0]-1.0)], [0,0] ] ]]) T=sage.gsl.ode.ode_solver() T.algorithm="rk8pd" f_1 = pg.f_1 j_1 = pg.j_1 pg.T=T python.execute("T.function=f_1") T.jacobian=j_1 python.execute("T.ode_solve(y_0=[1,0],t_span=[0,100],params=[10],num_points=1000)") python.execute(string.format("T.plot_solution(filename='%s')",graphout )) end \stopluacode \def\TestODE#1{% \ctxlua{test_ode("#1")}% \startcombination[2*1] {%foo \vbox{\hsize=8cm Consider solving the Van der pol oscillator $x''(t) +ux'(t)(x(t)^2-1)+x(t)=0 $ between $t=0$ and $t= 100$. As a first order system it is $x'=y$ $y'=-x+uy(1-x^2)$ Let us take $u=10$ and use initial conditions $(x,y)=(1,0)$ and use the \emphsl{\hbox{runga-kutta} \hbox{prince-dormand}} algorithm. }% }{\ss \ } {\externalfigure[#1][width=9cm]}{\ss Result for 1000 points} \stopcombination } \starttext \startTEXpage \TestODE{ode1.pdf} \stopTEXpage \stoptext
(other examples follows...)
ROOT (CERN)
ROOT -- an Oject-Oriented Data Analysis Framework -- is explained in
Wikipedia:
ROOT is an object-oriented program and library developed by CERN. It was originally designed for particle physics data analysis and contains several features specific to this field, but it is also commonly used in other applications such as astronomy and data mining.
For more infos, see here (here for python stuffs).
Under Linux installation is not difficult at all, so in this case I choose to not create a luatex-lunatic apart, as done above for sagemath.
See an example here .
ConTeXt mkIV examples
Here I will collect some tex snippets, just to show some ideas.
Python Imaging Library (PIL)
ROOT
This example shot how to literally embed original python source code .
We can do a bit better: separate python code from lua code .
Save this in test-ROOT1.py (so it's also easy to test) :
from ROOT import TCanvas, TGraph ,TGraphErrors,TMultiGraph from ROOT import gROOT from math import sin from array import array def run(filename): c1 = TCanvas("c1","multigraph",200,10,700,500) c1.SetGrid() # draw a frame to define the range mg = TMultiGraph() # create first graph n = 24; x = array('d',range(24)) data = file('data').readlines() for line in data: line = line.strip() y = array('d',[float(d) for d in line.split()]) gr = TGraph(n,x,y) gr.Fit("pol6","q") mg.Add(gr) mg.Draw("ap") #force drawing of canvas to generate the fit TPaveStats c1.Update() c1.Print(filename)
Here file 'data' is a 110 lines file with 24 floats values space separated,
ie
20.6000 19.4000 19.4000 18.3000 17.8000 16.1000 16.7000 21.1000 23.3000 26.1000 26.1000 27.2000 27.8000 28.3000 28.3000 27.2000 25.6000 22.8000 21.7000 21.7000 21.7000 21.7000 21.7000 21.7000 .
Now a tex file, with a simple layer in lua as interface for python:
Fontforge
In this example, we will use Metapost to draw a bezier curve of a glyph (Note: starting from Metapost 1.200 it is now possible to get the actual path drawing routines from a font glyph, so this example is only to show how to translate a path in metapost).
We will use 3-layer approach:
- a python layer that export a class,
- a lua layer to manage objects of this class
- a (con)TeX(t) layer that exports macros, because tex works very well with macros .
Let's start with python code, test-fontforge.py:
import sys import fontforge class simpledraw(object): def __init__(self,font_file): self.font = fontforge.open(font_file) def getcurve(self,letter): self.glname = letter res = dict() try : glyph_letter = [ g for g in self.font.glyphs() if g.glyphname == self.glname][0] except Exception ,e : res['err'] = str(e) return res cnt= glyph_letter.layers[1][0] res['is_quadratic'] = cnt.is_quadratic res['closed'] = cnt.closed res['points'] = [(p.x,p.y,"%i" %p.on_curve) for p in cnt ] res['design_size'] = self.font.design_size res['em'] = self.font.em return res def getmpostoutline(self,letter): res = self.getcurve(letter) path = '..'.join( [str((p[0],p[1])) for p in res['points'] if p[2] == '1'] ) return path def getmpostpoints(self,letter): res = self.getcurve(letter) path = [str((p[0],p[1])) for p in res['points'] if p[2] == '1'] return path def getmpostpointsSugar(self,letter): res = self.getcurve(letter) path = 'drawdot '.join( ["%s;" %str((p[0],p[1])) for p in res['points'] if p[2] == '1'] ) return 'drawdot ' +path if __name__ == '__main__': s = simpledraw("koeieletters.pfb") res = s.getmpostpointsSugar('C') print res
Note the '__main__' check, so we can test this class from python.
Next lua layer, which in this case is embed in a tex file:
\startluacode function testFontforge(fontfile,letter) require("python") testoutlines = python.import("test-fontforge") s = testoutlines.simpledraw(fontfile) g = s.getmpostoutline(letter) p = s.getmpostpointsSugar(letter) tex.sprint(tex.ctxcatcodes,"\\startMPcode") tex.sprint(tex.ctxcatcodes,"pickup pencircle scaled 1pt;") tex.sprint(tex.ctxcatcodes,string.format("draw %s .. cycle;",g) ) tex.sprint(tex.ctxcatcodes,"pickup pencircle scaled 8pt;") tex.sprint(tex.ctxcatcodes,string.format("%s",p) ) tex.sprint(tex.ctxcatcodes,"\\stopMPcode") end \stopluacode \def\Outline[#1]{% \getparameters[test][#1]% \ctxlua{testFontforge("\testfontfile", "\testletter")}% } \starttext \startTEXpage \Outline[letter={C}, fontfile={lmmono10-regular.otf}]% \Outline[letter={o}, fontfile={lmmono10-regular.otf}]% \Outline[letter={n}, fontfile={lmmono10-regular.otf}]% \Outline[letter={T}, fontfile={lmmono10-regular.otf}]% \Outline[letter={e}, fontfile={lmmono10-regular.otf}]% \Outline[letter={X}, fontfile={lmmono10-regular.otf}]% \Outline[letter={t}, fontfile={lmmono10-regular.otf}]% \stopTEXpage \stoptext
Here we use tex.sprint(tex.ctxcatcodes,"\\stopMPcode") to inject tex code (actually Metapost code) into TeX parser .
\Outline is the TeX layer: of course one can write \Outline and testFontforge in a different manner to avoid use of tex.sprint(..)
And this is the result:
...ok,it's not correct (why?), but it looks funny :)
Ghostscript
There are essentially 2 kind of use of ghostscript :
- convert an existing eps / ps file in pdf ;
- use a program in postscript that take an input, do something and make a ps output ( e.g. a barcode/label generator ).
For the first case, we consider an implementation of eps2pdf, being ps2pdf virtually the same .
Actually there is not a python binding of ghostscript, so we build a simple wrapper
using ctypes module.
import ctypes import sys class gs(object): def __init__(self): self.ierrors = dict() self.ierrors['e_unknownerror'] = -1 self.ierrors['e_dictfull'] = -2 self.ierrors['e_dictstackoverflow'] = -3 self.ierrors['e_dictstackunderflow'] = -4 self.ierrors['e_execstackoverflow'] = -5 self.ierrors['e_interrupt'] = -6 self.ierrors['e_invalidaccess'] = -7 self.ierrors['e_invalidexit'] = -8 self.ierrors['e_invalidfileaccess'] = -9 self.ierrors['e_invalidfont'] = -10 self.ierrors['e_invalidrestore'] = -11 self.ierrors['e_ioerror'] = -12 self.ierrors['e_limitcheck'] = -13 self.ierrors['e_nocurrentpoint'] = -14 self.ierrors['e_rangecheck'] = -15 self.ierrors['e_stackoverflow'] = -16 self.ierrors['e_stackunderflow'] = -17 self.ierrors['e_syntaxerror'] = -18 self.ierrors['e_timeout'] = -19 self.ierrors['e_typecheck'] = -20 self.ierrors['e_undefined'] = -21 self.ierrors['e_undefinedfilename'] = -22 self.ierrors['e_undefinedresult'] = -23 self.ierrors['e_unmatchedmark'] = -24 self.ierrors['e_VMerror'] = -25 self.ierrors['e_configurationerror'] = -26 self.ierrors['e_undefinedresource'] = -27 self.ierrors['e_unregistered'] = -28 self.ierrors['e_invalidcontext'] = -29 self.ierrors['e_invalidid'] = -30 self.ierrors['e_Fatal'] = -100 self.ierrors['e_Quit'] = -101 self.ierrors['e_InterpreterExit'] = -102 self.ierrors['e_RemapColor'] = -103 self.ierrors['e_ExecStackUnderflow'] = -104 self.ierrors['e_VMreclaim'] = -105 self.ierrors['e_NeedInput'] = -106 self.ierrors['e_Info'] = -110 self.libgspath = '/opt/luatex/luatex-lunatic/lib/libgs.so' self.OutFile = '' self.InFile = '' self.args = [] def appendargs(self,arg): if arg.find('-sOutputFile')>= 0: return if arg.find('-c quit')>= 0: return self.args.append(arg) def rawappendargs(self,arg): self.args.append(arg) def run(self): libgs = ctypes.CDLL(self.libgspath) exit_status = ctypes.c_int() code = ctypes.c_int(1) code1 = ctypes.c_int() instance = ctypes.c_void_p(None) exit_code = ctypes.c_int() code.value = libgs.gsapi_new_instance(ctypes.byref(instance), None) if code.value == 0 : libgs.gsapi_set_stdio(instance, None, None, None) self.args.insert(0,'') # if len(self.OutFile) > 0: self.args.append('-sOutputFile=%s' %self.OutFile) if len(self.InFile) > 0: self.args.append("%s" %self.InFile) self.args.append('-c quit') arguments = self.args # argc = ctypes.c_int(len(arguments)) argv = (ctypes.c_char_p * argc.value)(*arguments) code.value = libgs.gsapi_init_with_args(instance, argc, argv) code1.value = libgs.gsapi_exit(instance) if code.value == 0 or code.value == self.ierrors['e_Quit']: code.value = code1.value if code.value == self.ierrors['e_Quit']: code.value = 0 libgs.gsapi_delete_instance(instance) exit_status.value = 0 if __name__ == '__main__': pyctyps = gs() pyctyps.appendargs('-q') pyctyps.appendargs('-dNOPAUSE') pyctyps.appendargs('-dEPSCrop') pyctyps.appendargs('-sDEVICE=pdfwrite') pyctyps.InFile = 'test.eps' pyctyps.OutFile = 'test.pdf' pyctyps.run()
The tex code
\startluacode function testgs(epsin,pdfout) require("python") gsmodule = python.import("testgs") ghost = gsmodule.gs() ghost.appendargs('-q') ghost.appendargs('-dNOPAUSE') ghost.appendargs('-dEPSCrop') ghost.appendargs('-sDEVICE=pdfwrite') ghost.InFile = epsin ghost.OutFile = pdfout ghost.run() end \stopluacode \def\epstopdf#1#2{\ctxlua{testgs("#1","#2")}} \def\EPSfigure[#1]{%lazy way to load eps \epstopdf{#1.eps}{#1.pdf}% \externalfigure[#1.pdf]% } \starttext \startTEXpage \startcombination[2*1] {\EPSfigure[tiger]}{\ss tiger.eps} {\EPSfigure[golfer]}{\ss golfer.eps} \stopcombination \stopTEXpage \stoptext
Another example:
here we use a library to generate barcodes (see here) .
\startluacode function epstopdf(epsin,pdfout) require("python") gsmodule = python.import("testgs") ghost = gsmodule.gs() ghost.appendargs('-q') ghost.appendargs('-dNOPAUSE') ghost.appendargs('-dEPSCrop') ghost.appendargs('-sDEVICE=pdfwrite') ghost.InFile = epsin ghost.OutFile = pdfout ghost.run() end function barcode(text,type,options,savefile) require("python") gsmodule = python.import("testgs") barcode_string = string.format("%%!\n100 100 moveto (%s) (%s) %s barcode showpage" ,text,options,type) psfile = string.format("%s.ps",savefile) epsfile = string.format("%s.eps",savefile) pdffile = string.format("%s.pdf",savefile) temp = io.open(psfile,'w') temp:write(tostring(barcode_string),"\n") temp:flush() io.close(temp) ghost = gsmodule.gs() ghost.rawappendargs('-q') ghost.rawappendargs('-dNOPAUSE') ghost.rawappendargs('-sDEVICE=epswrite') ghost.rawappendargs(string.format('-sOutputFile=%s',epsfile)) ghost.rawappendargs('barcode.ps') ghost.InFile= psfile ghost.run() end \stopluacode \def\epstopdf#1#2{\ctxlua{epstopdf("#1","#2")}} \def\EPSfigure[#1]{%lazy way to load eps \epstopdf{#1.eps}{#1.pdf}% \externalfigure[#1.pdf]% } \def\PutBarcode[#1]{% \getparameters[bc][#1]% \ctxlua{barcode("\csname bctext\endcsname","\csname bctype\endcsname","\csname bcoptions\endcsname","\csname bcsavefile\endcsname" )}% \expanded{\EPSfigure[\csname bcsavefile\endcsname]}% } \starttext \startTEXpage \startcombination[2*2] {\PutBarcode[text={CODE 39},type={code39},options={includecheck includetext},savefile={TEMP1}]}{\ss code39} {\PutBarcode[text={CONTEXT},type={code93},options={includecheck includetext},savefile={TEMP2}]}{\ss code93} {\PutBarcode[text={977147396801},type={ean13},options={includetext},savefile={TEMP3}]}{\ss ean13} {\PutBarcode[text={0123456789},type={interleaved2of5},options={includecheck includetext},savefile={TEMP4}]}{\ss interleaved2of5} \stopcombination \stopTEXpage \stoptext
R
R is a language and environment for statistical computing and graphics (see here) .
RPy is a very simple, yet robust, Python interface to the R Programming Language (see here) .
As example, let's try to plot a discrete distribution of probability for a set of pseudorandom number (around 100000 samples) .
import rpy2.robjects as robjects import rpy2.rinterface as rinterface class density(object): def __init__(self,samples,outpdf,w,h,kernel): self.samples = samples self.outpdf= outpdf self.kernel = kernel self.width=w self.height=h def run(self): r = robjects.r data = [int(k.strip()) for k in file(self.samples,'r').readlines()] x = robjects.IntVector(data) r.pdf(file=self.outpdf,width=self.width,height=self.height) z = r.density(x,kernel=self.kernel) #r.plot(z[0],z[1],xlab='',ylab='',xlim=robjects.IntVector([0,2**16-1])) r.plot(z[0],z[1],xlab='',ylab='') r['dev.off']() if __name__ == '__main__' : dens = density('u-random-int','test-001.pdf',10,7,'o') dens.run()
\startluacode function testR(samples,outpdf,w,h,kernel) require("python") testR = python.import("test-R") dens = testR.density(samples,outpdf,w,h,kernel) dens.run() end \stopluacode \def\plotdenstiy[#1]{% \getparameters[R][#1]% \expanded{\ctxlua{testR("\Rsamples","\Routpdf",\Rwidth,\Rheight,"\Rkernel")}}% } \setupbodyfont[sans,10pt] \starttext \startTEXpage \plotdenstiy[samples={u-random-int},outpdf={test-001.pdf},width={10},height={7},kernel={o}] \setupcombinations[location=top] \startcombination[1*2] {\vbox{\hsize=400bp This is a density plot of around {\tt 100 000} random numbers between $0$ and $2^{16}-1$ generated from {\tt \hbox{/dev/urandom}}}}{} {\externalfigure[test-001.pdf][width={400bp}]}{} \stopcombination \stopTEXpage \stoptext
And here is the plot