User:Luigi.scarso/luatex lunatic

From Wiki
Jump to navigation Jump to search

Introduction

!! W A R N I N G !!

!! THIS CODE IS HIGHLY EXPERIMENTAL !!

!! THIS CODE WORKS ONLY UNDER LINUX !!

luatex_lunatic is a modification of lua side of luatex to host a python interpreter inside lua (see lunatic-python) .

I have made a set of patches because :

  • by design, lua in luatex doesn't permit dynamic loading ("Dynamic loading of .so and .dll files is disabled on all platforms." see "LUA changes" in manual/luatexref-t.pdf of src and/or svn dist.)
  • to prevent symbols collisions between luatex and an arbitrary library

Having a python interpreter hosted in luatex can make easy to use existing python's bindings (also Python releases includes at least from 2.5 a ctypes module that permit a binding to a .so without using SWIG or similar, similar to lua's loadlib) .

As general rule, I want the smallest set of patches ; so, for example, I have choose to not use a system libpng, even if it's easy to modify build.sh.linux to do so.

Also, I'm not concerned about portability: so I will talk only about Linux .

Please note that luatex_lunatic was born to answer to (my) question "Can I apply lunatic to luatex ?" and nothing else.

Every others meanings can be interestring, but are not mine , and I will not partecipate to any discussion about that.

How to

I prefear to put a bash script that can help to compile a luatex-lunatic binary. I have put all patches files here It's clear that some skills are needed, so I hope that this is barrier to avoid useless questions . Of course, I will try to correct all errors .

The trick is "hide all, then unhide what is needed" : in build.sh.linux change 

CFLAGS="-g -O2 -Wno-write-strings"
CXXFLAGS=$CFLAGS

in 

CFLAGS="-g -O2 -Wno-write-strings -fvisibility=hidden"
CXXFLAGS="$CFLAGS -fvisibility-inlines-hidden"

and build all. After that, make only what is needed (lua51) and re-compile luatex . 


##
## Sec. 0
## Safe exit 
## uncomment exit to go on
exit

##
##
## Sec. 1
## Setup
## 
##
if [! -d Python-2.6.1 ];
then
$HOME_LUN="/opt/luatex/luatex-lunatic"
svn checkout svn://scm.foundry.supelec.fr/svn/luatex/tags/beta-0.35.0
#svn checkout svn://scm.foundry.supelec.fr/svn/luatex/trunk
bzr branch lp:lunatic-python
wget http://www.python.org/ftp/python/2.6.1/Python-2.6.1.tar.bz2
tar -xjvf Python-2.6.1.tar.bz2 
cd  Python-2.6.1
./configure --prefix=$HOME_LUN --enable-unicode=ucs4 --enable-shared
make && make install
cp python2.6.conf /etc/ld.so.conf.d
ldconfig ;
fi
##
## install setuptools
if [ ! -x bin/easy_install ];
then
echo "Please install setuptools"
echo "from  http://pypi.python.org/pypi/setuptools"
echo "in $HOME_LUN" 
echo "Please, pay attention to use $HOME_LUN/bin/python , not system python !" 
echo "Eventually modify first lines of setuptools to accomodate this,"
echo "or modify \$PATH=$HOME_LUN/bin:\$PATH" 
exit;
fi
##
##
## Sec 2
## Patch and build luatex
##
##
cd $HOME_LUN
#ln -s trunk luatex
##
## build lunatic
##
ln -s beta-0.35.0 luatex
patch -N --backup <0000_lunatic-python_setup.py.patch  lunatic-python/setup.py
cd lunatic-python
../bin/python setup.py build
../bin/python setup.py install --root=/ --prefix=$HOME_LUN --install-script=$HOME_LUN/bin
cd $HOME_LUN
##
## This patch set 
## CFLAGS="-g -O2 -Wno-write-strings -fvisibility=hidden"
## to avoid symbols conflicts
patch -N --backup <0500_build.sh.linux.patch luatex/build.sh.linux
##
## These patches 
## permits dynamic loading of shared libs.
##
patch -N --backup <1000_luatex_src_libs_lua51_loadlib.c.patch luatex/src/libs/lua51/loadlib.c
patch -N --backup <2000_luatex_src_libs_lua51_Makefile.patch  luatex/src/libs/lua51/Makefile
patch -N --backup <3000_luatex_src_texk_web2c_configure.patch luatex/src/texk/web2c/configure
##
## Build luatex
##
cd luatex
./build.sh.linux &> out 
cd $HOME_LUN
mkdir tests
##
## "Unhide" visibility of lua51
##
cd luatex-lunatic/luatex/build/libs/lua51
make -B posix
##
## Re-compile luatex
##
cd $HOME_LUN
cd luatex-lunatic/luatex/build/texk/web2c
g++ -o luatex  luatexini.o luatex0.o luatex1.o luatex2.o luatex3.o luatexextra.o luatex-pool.o   luatexdir/libpdf.a ../../libs/libpng/libpng.a ../../libs/zlib/libz.a ../../libs/xpdf/xpdf/libxpdf.a ../../libs/xpdf/goo/libGoo.a ../../libs/xpdf/fofi/libfofi.a ../../libs/md5/md5.o ../../libs/obsdcompat/libopenbsd-compat.a ../../libs/lua51/liblua.a ../../libs/slnunicode/slnunico.o  ../../libs/luazip/src/luazip.o ../../libs/zziplib/zzip/libzzip.a ../../libs/luafilesystem/src/lfs.o ../../libs/luasocket/src/socket.a ../../libs/luapeg/lpeg.o ../../libs/luamd5/md5lib.o ../../libs/luamd5/md5.o  ../../libs/luazlib/lgzip.o ../../libs/luazlib/lzlib.o ../../libs/luafontforge/libff.a ../../libs/luaprofiler/libprofiler.a mpdir/lmplib.o mpdir/.libs/libmplib.a   lib/lib.a ../kpathsea/.libs/libkpathsea.a -lm   -nodefaultlibs -Wl,-Bstatic -lstdc++ -Wl,-Bdynamic -ldl -lm -lgcc_eh -lgcc -lc -lgcc_eh -lgcc -Wl,-E -ldl -lreadline -lhistory -lncurses
##
##

The last line is:

g++ -o luatex luatexini.o luatex0.o luatex1.o luatex2.o luatex3.o luatexextra.o luatex-pool.o luatexdir/libpdf.a ../../libs/libpng/libpng.a ../../libs/zlib/libz.a ../../libs/xpdf/xpdf/libxpdf.a ../../libs/xpdf/goo/libGoo.a ../../libs/xpdf/fofi/libfofi.a ../../libs/md5/md5.o ../../libs/obsdcompat/libopenbsd-compat.a ../../libs/lua51/liblua.a ../../libs/slnunicode/slnunico.o ../../libs/luazip/src/luazip.o ../../libs/zziplib/zzip/libzzip.a ../../libs/luafilesystem/src/lfs.o ../../libs/luasocket/src/socket.a ../../libs/luapeg/lpeg.o ../../libs/luamd5/md5lib.o ../../libs/luamd5/md5.o ../../libs/luazlib/lgzip.o ../../libs/luazlib/lzlib.o ../../libs/luafontforge/libff.a ../../libs/luaprofiler/libprofiler.a mpdir/lmplib.o mpdir/.libs/libmplib.a lib/lib.a ../kpathsea/.libs/libkpathsea.a -lm -nodefaultlibs -Wl,-Bstatic -lstdc++ -Wl,-Bdynamic -ldl -lm -lgcc_eh -lgcc -lc -lgcc_eh -lgcc -Wl,-E -ldl -lreadline -lhistory -lncurses

Python packages

  • numpy ✔
  • scipy ✔
  • matplot ✔
  • odfpy ✔
  • PIL, python imaging library ✔ PIL
  • TO FIX ; pygegl (I like its syntax, but need too much and it's easy only in python)
  • TODO: binding of libtiff with ctypes (or use gdal ?)
  • TODO :gle (for Massimiliano "Max" Dominici, GUIT)
  • TODO : using the binding of VIPS

Bindings

  • ghostscript 8.64 ✔
  • graphviz 2.22.0 ✔
  • ImageMagick-6.4.9 with pythonmagickwand ✔
  • fontforge 20090224 ✔ Useful to check symbols collision, and if one want to play with the last fontforge, eg to draw the outline of a glyph .
  • R-2.8.1 with rpy2-2.0.3 (For Maurizio "Mau" Himmelman , GUIT) ✔ need an example with output in pdf
  • quantlib 0.9.7 ✔ need an example with output in pdf
  • dbxml-2.4.16 ✔

Dedicated systems

SageMath

Given that sagemath is rooted on a CPtyhon , we can try to "use sagemath as luatex plugin" instead of "use latex as external process of sagemath".

I don't want to mix with luatex-lunatic, so I put all stuffs under lunatic-python-SAGEMATH . 

$HOME_LUN=/opt/luatex/luatex-lunatic
cd $HOME_LUN
cp -r lunatic-python lunatic-python-SAGEMATH
cd lunatic-python-SAGEMATH
##
## We prepare new bridge
##
rm -r build
$HOME_LUN/sage/local/bin/python setup.py build
cd $HOME_LUN
mkdir tests-SAGEMATH
##
## I have already installed prev. python.so, I don't want mess things
##
ln -s ../lunatic-python-SAGEMATH/build/lib.linux-i686-2.5/python.so
ln -s  ../luatex/build/texk/web2c/luatex
##
## now we need to setup sagemath
##
cd $HOME_LUA/sage/local/bin
. sage-env
cd $HOME_LUA

We need also a first run of sagemath, if you never run it: 

cd sage
./sage
##
## exit from sage
##

That is .

Now a simple test: 

function test_SAGEMATH()
  require("python")
  python.execute("from sage.all_cmdline import *")
  dir = python.eval("dir()")
  print(dir)    
end
test_SAGEMATH()

$>luatex test-SAGEMATH.lua

You should see something like this:

['AA', 'AbelianGroup', 'AbelianGroupElement', 'AbelianGroupMap', 'AbelianGroupMorphism', 'AbelianGroupMorphism_id', 'AbelianGroup_class', 'AbelianGroup_subgroup', 'AbelianGroups', 'AbelianMonoids', 'AbelianSemigroups', 'AbelianVariety', 'AffineGeometryDesign', 'AffineSpace', 'Algebra', 'AlgebraElement', 'AlgebraIdeals', 'AlgebraModules', 'AlgebraicField', 'AlgebraicNumber', 'AlgebraicReal', 'AlgebraicRealField', 'Algebras', 'AllCusps', 'AllExactCovers', 'Alphabet', 'AlphabeticStrings', 'AlternatingGroup', 'AlternatingSignMatrices', 'AntichainPoset', 'Arrangements', 'AtkinModularCorrespondenceDatabase', 'AtkinModularPolynomialDatabase', 'AugmentedLatticeDiagramFilling', 'Axiom', 'BCHCode', 'Bessel', 'BinaryGolayCode', 'BinaryQF', 'BinaryQF_reduced_representatives', 'BinaryReedMullerCode', 'BinaryStrings', 'BinaryTree', 'BipartiteGraph', 'BlockDesign', 'BlockDesign_generic', 'BooleanLattice', 'BooleanPolynomialRing', 'CC', 'CDF', 'CFF', 'CIF'

.....

true', 'ttest', 'tuples', 'twinprime', 'twisted', 'two_squares', 'ultraspherical', 'uniform', 'union', 'uniq', 'unordered_tuples', 'unpickle_function', 'unset_verbose_files', 'unsigned_infinity', 'upgrade', 'valuation', 'var', 'vecsmall_to_intlist', 'vector', 'verbose', 'version', 'vert_to_ieq', 'victor_miller_basis', 'view', 'volume_hamming', 'vonmisesvariate', 'walltime', 'walsh_matrix', 'wave', 'weakref', 'weibullvariate', 'wiki', 'wiki_create_instance', 'with_statement', 'word_problem', 'words', 'wrapper', 'wronskian', 'x', 'xgcd', 'xinterval', 'xlcm', 'xmrange', 'xmrange_iter', 'xsrange', 'zero', 'zero_matrix', 'zeta', 'zeta_symmetric', 'zeta_zeros']

ROOT (CERN)

ROOT -- an Oject-Oriented Data Analysis Framework --
From Wikipedia:
ROOT is an object-oriented program and library developed by CERN. It was originally designed for particle physics data analysis and contains several features specific to this field, but it is also commonly used in other applications such as astronomy and data mining.

For more infos, see here (here for python stuffs).

Under Linux installation is not difficult at all; see an example here .

ConTeXt mkIV examples

Here I will collect some tex snippets, just to show some ideas.

Python Imaging Library (PIL)

\startluacode
function testPIL(imageorig,imagesepia)
  require("python")
  PIL_Image = python.import("PIL.Image") 
  PIL_ImageOps = python.import("PIL.ImageOps") 
  python.execute([[
def make_linear_ramp(white):
    ramp = []
    r, g, b = white
    for i in range(255):
        ramp.extend((r*i/255, g*i/255, b*i/255))
    return ramp
]])
        
    -- make sepia ramp (tweak color as necessary)
    sepia = python.eval("make_linear_ramp((255, 240, 192))") 
    im = PIL_Image.open(imageorig)
    
    -- convert to grayscale
    if not(im.mode == "L")
     then 
        im = im.convert("L")
    end
    
    --  optional: apply contrast enhancement here, e.g.
    im = PIL_ImageOps.autocontrast(im)

    --  apply sepia palette
    im.putpalette(sepia)

    --  convert back to RGB so we can save it as JPEG
    --  (alternatively, save it in PNG or similar)
    im = im.convert("RGB")

    im.save(imagesepia)
end
\stopluacode

\def\SepiaImage#1#2{%
\ctxlua{testPIL("#1","#2")}%
\startcombination[2*1]
{\externalfigure[#1]}{\ss Orig.}
{\externalfigure[#2]}{\ss Sepia}
\stopcombination
}


\starttext
\startTEXpage
\SepiaImage{lena.jpg}{lena-sepia.jpg}
\stopTEXpage
\stoptext

TestPIL.jpg


ROOT

This example shot how to literally embed original python source code . 

\startluacode
function test_ROOT(filename)
  require("python")
  pg = python.globals()

  python.execute([[
def run(filename):
    from ROOT import TCanvas, TGraph
    from ROOT import gROOT
    from math import sin
    from array import array


    gROOT.Reset()

    c1 = TCanvas( 'c1', 'A Simple Graph Example', 200, 10, 700, 500 )

    c1.SetFillColor( 42 )
    c1.SetGrid()

    n = 20
    x, y = array( 'd' ), array( 'd' )

    for i in range( n ):
          x.append( 0.1*i )
          y.append( 10*sin( x[i]+0.2 ) )

    gr = TGraph( n, x, y )
    gr.SetLineColor( 2 )
    gr.SetLineWidth( 4 )
    gr.SetMarkerColor( 4 )
    gr.SetMarkerStyle( 21 )
    gr.SetTitle( 'a simple graph' )
    gr.GetXaxis().SetTitle( 'X title' )
    gr.GetYaxis().SetTitle( 'Y title' )
    gr.Draw( 'ACP' )
    c1.Update()
    c1.Print(filename)
]])
  run = pg.run
  run(filename)
end
\stopluacode

\starttext
\startTEXpage
\ctxlua{test_ROOT("testsin.pdf")}
\rotate[rotation=90]{\externalfigure[testsin.pdf][width=5cm]}
\stopTEXpage
\stoptext
Testsin.jpg

We can do a bit better: separate python code from lua code .
Save this in test-ROOT1.py 

from ROOT import TCanvas, TGraph ,TGraphErrors,TMultiGraph
from ROOT import gROOT
from math import sin
from array import array

def run(filename):
    c1 = TCanvas("c1","multigraph",200,10,700,500)
    c1.SetGrid()

    # draw a frame to define the range
    mg = TMultiGraph()
    #   create first graph
    n = 24;
    x = array('d',range(24))
    data = file('data').readlines()
    for line in data:
        line = line.strip()
        y  = array('d',[float(d) for d in line.split()])
        gr =  TGraph(n,x,y)
        gr.Fit("pol6","q")
        mg.Add(gr)

    mg.Draw("ap")

    #force drawing of canvas to generate the fit TPaveStats
    c1.Update()
    c1.Print(filename)

Here file 'data' is a 110 lines file with 24 floats values space separated ie
20.6000 19.4000 19.4000 18.3000 17.8000 16.1000 16.7000 21.1000 23.3000 26.1000 26.1000 27.2000 27.8000 28.3000 28.3000 27.2000 25.6000 22.8000 21.7000 21.7000 21.7000 21.7000 21.7000 21.7000 .
Now a tex file, wih a simple layer in lua as interface for python: 

\startluacode
function test_ROOT(filename)
  require("python")
  test = python.import('test-ROOT1')
  test.run(filename)
end
\stopluacode

\starttext
\startTEXpage
\ctxlua{test_ROOT("data.pdf")}
\rotate[rotation=90]{\externalfigure[data.pdf]}
\stopTEXpage
\stoptext
Test-ROOT1.jpg
Retrieved from "https://wiki.contextgarden.net/index.php?title=User:Luigi.scarso/luatex_lunatic&oldid=9496"