TensorFlow 1.4.0 - Stars Growing (from TF 0.10 "raindrops" Tutorial)
I like any simulations - even simple ones. This one is great, because it runs and displays images as it goes along. Even with Jupyter, I have yet to figure out a way to hold the screen steady, and not have it flicker as the simulation runs. In Jupyter, you can clear the screen each time - but that makes a big flicker.
What I do here is just display a created .jpg to the screen every few hundred iterations. If you get everything installed, and run this, you should see six images be generated in real-time, as the simple simulation unfolds. The final image should look something like the one on the right. It turns out in Python, it is non-trivial to get the image to actually display from within a loop, right now. My early attempts at doing this and running it using plain Python (not Jupyter or Interactive Python), would just display a message indicating an image had been created. It would not show the image.
To show the image programatically, I used "imageio" to create the image, and then PIL (the PIllow Library - the updated version of the Python Image Library) to load and show() the image.
This program just confirms TensorFlow is installed correctly, and does it's math ok. The program takes about 37 seconds to run on my Intel Core-i3 64-bit Linux box, roughly 5 or six seconds between each image display. The Python PIP library uses "ImageMagick" to render images from temporary files as it runs, but it does not require user input to click-close the image (like using os.system to call "display image.jpg" would), so you see 6 images overlaid on the screen, after it completes.
#-------------------------------------------------------------------------------
# --- the Tensorflow LaPlace Image example (Uses PIL, and IPython)
# --- updated for TensorFlow 1.4.0 running on CentOS-7.4 & Python 2.7.14
# compiled (configured, actually)
with the "--enable-unicode=ucs4" option
#
(Python compile default is ucs2)
#
(which caused TensorFlow 1.4 to)
#
(fail to load. Building Python )
#
(with ucs4, => pip can install )
#
(TensorFlow 1.4.0 successfully.)
# --- Import libraries for simulation
import tensorflow as tf
import numpy as np
import scipy.misc
import imageio
import os
import PIL
# --- Import for visualization and jpeg
encoder
import matplotlib
matplotlib.rcParams["backend"]= 'TkAgg'
from matplotlib import pyplot as plt
# from PIL import Image, ImageDraw
from io import BytesIO
from IPython.display import clear_output, Image, display
# --- function for displaying state of the pond's surface as an image
def DisplayArray(a, fmt='jpeg', rng=[0,1]):
# """Display an array as a picture. """
a = (a - rng[0])/float(rng[1] - rng[0])*37
a = np.uint8(np.clip(a,
0, 255))
f = BytesIO()
# --- this is the cool, realtime thing that runs in Jupyter-IPython Notebook
PIL.Image.fromarray(a).save(f,fmt)
clear_output(wait = True)
# display(Image(data=f.getvalue()))
# --- write the image
# --- this is my stuff to write the simulation images to .jpg files
scipy.misc.imsave("tensor.jpg", a)
image = PIL.Image.open("tensor.jpg")
image.show()
# clear_output(wait=True)
# --- this line below doesn't work outside of the Jupyter environment...
# display(Image(data=f.getvalue()))
#
def DisplayArrayToFile(a, fmt='jpeg', rng=[0,1]):
# """Display an
array as a picture to a file... """
a = (a - rng[0])/float(rng[1] - rng[0])*37
a = np.uint8(np.clip(a, 0, 255))
f = BytesIO()
# --- this is the cool, realtime thing that runs in Jupyter-IPython Notebook
PIL.Image.fromarray(a).save(f,fmt)
#clear_output(wait = True)
#display(Image(data=f.getvalue()))
# --- write the image
# --- this is my stuff to write the simulation images to .jpg files
#scipy.misc.imsave
("tensor_new.jpg", a)
imageio.imwrite("tensor_new.jpg", a)
image = PIL.Image.open("tensor_new.jpg")
image.show()
#clear_output(wait=True)
#display(Image(data=f.getvalue()))
#
# --- make interactive session for messing around - can use regular session also
sess = tf.InteractiveSession()
# sess = tf.Session()
# --- computational functions go
here... once we get jpeg pic working
def make_kernel(a):
"""Transform a 2D array into a convolutional kernel """
a = np.asarray(a)
a = a.reshape(list(a.shape) + [1,1])
return tf.constant(a, dtype=1)
def simple_conv(x, k):
""" A simplified 2D convolutional operation """
x = tf.expand_dims(tf.expand_dims(x, 0), -1)
y = tf.nn.depthwise_conv2d(x, k, [1, 1, 1, 1], padding='SAME')
return y[0, :, :, 0]
def laplace(x):
"""Compute the 2D laplacian of an array """
laplace_k = make_kernel([[0.5, 1.0, 0.5],
[1.0, -6., 1.0],
[0.5, 1.0, 0.5]])
return simple_conv(x, laplace_k)
# --- Define the PDE - the pond is a perfect 500x500 square
N = 500
# --- now, we create some raindrops
# --- Initial Conditions -- some rain drops hit the pond
# --- set everything to zero
u_init = np.zeros([N,
N], dtype=np.float32)
ut_init = np.zeros([N, N], dtype=np.float32)
# Some material accretion occurs (raindrops hit pond) at random points
for n in range(40):
a,b = np.random.randint(0, N, 2)
u_init[a,b] = np.random.uniform()
# --- Create and Display the jpeg image...
DisplayArray(u_init, rng=[-0.1, 0.1])
# Parameters
# eps -- time resolution
# damping -- wave damping
eps = tf.placeholder(tf.float32, shape=())
damping = tf.placeholder(tf.float32,
shape=())
# --- Create vaiables for simulation state
U = tf.Variable(u_init)
Ut = tf.Variable(u_init)
# --- Discretized PDE update rules
U_ = U + eps * Ut
Ut_ = Ut + eps * (laplace(U) - damping * Ut)
# --- Operation to update the state
step = tf.group(
U.assign(U_),
Ut.assign(Ut_))
# --- Run the simulation forward with a simple FOR loop.
# --- Initialize state to initial conditions
# sess=tf.InteractiveSession
none of these 4 work...
# with tf.Session().as_default() as sess:
# sess.as_default()
# with sess.as_default():
tf.global_variables_initializer().run(session=sess)
# --- Run 1470 steps of PDE
for i in range(1470):
# Step simulation (damping was 0.04, I made it negative .14)
with sess.as_default(): step.run( {eps: 0.03, damping: -0.14})
if (i == 400) :
with sess.as_default(): DisplayArray(U.eval(),
rng=[-0.1, 0.1])
if (i == 700) :
with sess.as_default(): DisplayArray(U.eval(), rng=[-0.1, 0.1])
if (i == 1200) :
with sess.as_default(): DisplayArray(U.eval(),
rng=[-0.1, 0.1])
#
# --- to show each iteration...
# with sess.as_default(): DisplayArray(U.eval(), rng=[-0.1, 0.1])
# --- Ok, we are done...
with sess.as_default(): DisplayArray(U.eval(), rng=[-0.1, 0.1])
with sess.as_default(): DisplayArrayToFile(U.eval(), rng=[-0.1, 0.1])
print "Done."
#--------------- done ------------------
Example Python Program to Create Fully-Connected Graph
#--- draw fully connected graph example using Tck/Tk
import numpy as np
import _tkinter
# --- turn off warnings
import warnings
warnings.filterwarnings('ignore')
import matplotlib
matplotlib.rcParams["backend"] = 'TkAgg'
import matplotlib.pyplot
as plt
import networkx as nx
# --- ask user for size of graph to draw...
print
" "
print "Create Fully-Connected Graph."
print " "
gsize = int(input("Enter
the number of nodes to draw: "))
gvec = []
for i in range(gsize):
gvec.append(i+1)
g = nx.Graph()
# --- g = nx.cycle_graph(1)
g.add_nodes_from(gvec)
# A = nx.adjacency_matrix(G)
# --- gsize = 7
edgectr = 0
for a in range ( 1, (gsize+1)):
for b in range ((a+1), (gsize+1)):
g.add_edge(a, b)
edgectr = edgectr+1
# --- print "A is: ",a, "
B is: ",b
# --- g.add_edge(1, 7)
nx.draw_networkx(g)
print
"Drawing fully-connected graph, with: ",gsize," verticies"
print "Total number of links created was: ",edgectr
print "... (close graph-window
to exit...) "
plt.title("Fully Connected Graph - Nodes: "+str(gsize))
plt.show()
# --- done.
This is a .jpg image which simulates Moire patterns, and was created on a MacBook, using a Python program and the scipy and numpy (the scientific and numeric Python libraries) and PIL (the Python Image Library, but now actually called Pillow). This image was rendered using the Preview utility of the MacOs. The Python program that generated it is shown below.
Example Python Program to Create Simulated Moire Pattern
#
# --- Make and Show immediately a Moire pattern as a .jpg file
#
# --- For now, you start Python and import this file with: import MakeShowMoire
# --- or use: execfile ("MakeShowMoire.py")
#
# --- import modules we need
from PIL import Image, ImageDraw
import
numpy as np
import scipy.misc
#
# --- Ok, lets go..
width = 1020
height = 670
channels = 3
#
img = np.zeros((height, width, channels), dtype=np.uint8)
xx, yy = np.mgrid[:height, :width]
circle
= .2 * ( (xx - 1000) ** 2 + (yy + 1000) ** float(1.9795) )
print "Generating Image with shape y: ", img.shape[0], " x: ", img.shape[1], " ... "
#print "xx is: ", xx
#print "yy is: ", yy
#
for y in range(img.shape[0]):
for x in range(img.shape[1]):
r, g, b = circle[y][x], circle[y][x], circle[y][x]
img[y] [x] [0] = r
img[y] [x] [1] = g
img[y] [x] [2] = int(y / 2.7)
#
# --- now, we have made the image,
lets save it
scipy.misc.imsave("testimg.jpg", img)
#
# --- Since we have saved
it, we should be able to display it
image = Image.open("testimg.jpg")
image.show()
Code Examples: (Code is experimental, for use-at-own-risk, of course. )
Code examples:
Tck/Tk "CompareMarketNet.tcl" Evaluation Program
This simple program just processes the evaluation cases (same format and name of the original training cases used to train the network), and calculates a simple "co-efficient of accuracy", which is just the percentage of network outputs that successfully match the target. Since the network trains to, and also outputs signed boolean values, any value in the range of the absolute value of .8 or less is counted as a zero, for evaluation purposes. If the network generates a -.723 or a +.6824, that gets evaluated as a zero. If the network generates a .9124, that gets evaluated as a +1.0 boolean, and so on.
#
# --- run a compare between MarketNet "tcasetab.txt" Training set and trained network output
# --- also, calculate
a "coefficient of accuracy", basically the % right
#
#
proc compareMarketNet {} {
puts "MarketNet Network: Target versus Output from network... "
set rightctr 0
set totctr 0
#
loop i 0 [MNTraining size] {
uts_presentExample MarketNet MNTraining $i
uts_activateNet MarketNet
set cdate [MNTraining tag $i]
set output [lindex [MarketNet.Output.0 configure -output]
2]
set target [lindex [MarketNet.Output.0 configure -target] 2]
puts "$cdate :: MarketNet target: $target,
MarketNet output: $output"
set rndoutput round($output)
set rndtarget round($target)
set absvalout abs($output)
if {[expr [expr $rndoutput==$rndtarget] || [expr [expr $target==0.0] && [expr $absvalout<=0.80]]] } {
set rightctr [expr $rightctr+1]
}
if {$target != -999}
{
set totctr [expr $totctr+1]
}
}
puts "--- < Done > --- "
puts "Number Right: $rightctr"
puts "Total Cases : $totctr"
puts "Coeff.Accur : [expr 1.0 * $rightctr/$totctr]"
}
Building Neural Networks with Xerion:
The example here is for defining a Neural Network using Xerion, loading the training cases, and defining a tcl/tk program to plot the actual vs. predicted. Network is a 30-40-1 topology, the current MarketNet example used to predict whether the 5-day ahead price will be up up more than 1% (output = +1), no significant change (output = zero), or down more than 1% (output = -1). You can download Xerion and tcl/tk (and the various sub-components needed, such as tclX, itcl and tkmxm) from url: http://ftp.cs.toronto.edu/dist/xerion/
To compile Xerion, you may have to make a number of tweaks to get it to run on your local flavour of Linux. If you are using modern C compilers (I used an old gcc 4.3.0, circa 2008), you will need to include a varargs.h function, to allow variable number of args be passed. I used an old Xenix header, from a little-known left-coast software firm called Microsoft...). Here is a url to a copy of it, from the Arecibo Observatory website http://www.naic.edu/~phil/hardware/vertex/sharemegsvertex/ocu/pcr/mscv51/include/VARARGS.H This code snippet is dated 1985-1988, and has a Microsoft copyright stmt in it. Without the copyright notice, it is nine lines of code (8 lines, if you don't count the whitespace.) You will need to include a varargs.h into order to get Xerion to compile with a modern gcc.
Defining the Network, Loading Training Cases: (file: mktnet3040.in).
This file can be "sourced" into Xerion at the Xerion command line (at "bp-> " prompt), or from the Xerion gui screen, under the "File" option. It is basically bash + tcl/tk code.
###################################################################################
# A single hidden layer (40 units) net for Cross-sectional Time Segment MarketNet #
#
#
# Training Cases built by MAKECASE, an APL workspace, with function: MAKECASE #
# Database of Market Prices in: /home/cas/MARKET/DATA/timesr1.sf
#
# File "tcasetab.txt" created in ../MARKET/DATA, than copied to #
# directory
/home/cas/marketAI
#
#
#
#
#
# '$Id: 2017/05/17 mlangdon Exp $'
#
###################################################################################
set net MarketNet ;
catch {
$net delete ;
}
# --- Create a network with 30 inputs, 40 hidden units, 1 output unit
# --- this network will use TANH activation functions
#
uts_simpleNet $net 30 40 1 ;
bp_groupType MarketNet.Hidden {HIDDEN TANH}
bp_groupType MarketNet.Output {OUTPUT TANH}
random seed 3 ;
uts_randomizeNet MarketNet ;
# Add a new example set for training
set exampleSet "MNTraining" ;
catch {
$exampleSet delete ;
}
uts_exampleSet $exampleSet ;
uts_loadExamples $exampleSet
/home/cas/marketAI/tcasetab.txt
#
# Procedure for plotting network output and target
#
proc plotValues { net exampleSet} {
set t [open "target.dat"
w] ;
set o [open "output.dat" w] ;
set cr \r;
set blk " ";
set endnum [$exampleSet size];
loop idx
0 $endnum {
uts_presentExample $net $exampleSet $idx ;
uts_activateNet $net ;
regexp {([0-9]+)} [$exampleSet tag current] ddate ;
# if { $idx >= $strpnt } {
puts $o "$idx $blk [lindex [$net.Output.0 configure -output] 2] $blk $ddate $cr";
puts $t "$idx $blk [lindex [$net.Output.0
configure -target] 2] $blk $ddate $cr";
# }
}
close $t ;
close $o ;
set command {
set terminal X11 ;
set grid ;
set title "** Scaled ** MarketNet: CM Close in Boolean Jumpdeltas - Daily"
set xlabel "Date Index Position"
;
set ylabel "CM JumpDeltas " ;
set style data lines
#--- old format: set data style lines
#--- set yrange [.50:.800]
plot "target.dat" title "Actual CM", \
"output.dat" title "Predicted CM"
pause -1 "Hit Return to quit, and run
Postscipt prt to outfile" ;
set term postscript ;
set output "outfile" ;
plot "target.dat" title "Actual CM", \
"output.dat" title "Predicted CM"
pause -1 "Hit return to quit" ;
}
exec echo $command > plot.gnu
exec gnuplot
plot.gnu >@ stdout 2>@ stdout ;
}
< This file is "compareMarketNet.tcl", and is used to compare training target and actual network output>
#
# --- run a compare between MarketNet
"tcasetab.txt" Training set and trained network output
proc compareMarketNet {} {
puts "MarketNet Network: Target versus Output from network... "
loop i 0 [MNTraining size] {
uts_presentExample MarketNet MNTraining $i
uts_activateNet MarketNet
set cdate [MNTraining tag $i]
set output [lindex [MarketNet.Output.0 configure -output]
2]
set target [lindex [MarketNet.Output.0 configure -target] 2]
puts "$cdate :: MarketNet target: $target, MarketNet output: $output"
}
puts "---
< Done > --- "
}
Python & TensorFlow & PIL (Python Image Library) to display the exploding stars example.
Second piece of the code is the "raindrops" laplace PDE (partial differential equations) simulation, using Python 2.7 and TensorFlow, Numpy, Scipy and PIL (actually Pillow), on a MacBook. (See screen image from Macbook below.) The code is from Google TensorFlow tutorials. The original code uses Python Notebooks, Jupyter, and some other Python extentions, so the simulation can be seen to evolve in real-time - which is of course really cool. But it is also nice to get something actually running, just to checkpoint one's progress. This code below will produce the five images shown in the image to the right, and will confirm that you have the baseline configuration needed to get TensorFlow running on your Mac OSX. I used Python 2.7.11, because I had it there already. The MacBook OSX version is Yosemite (OSX 10.10.5). TensorFlow needs Python 2.7.x, at a minimum. (The MacBook came with Python 2.6 installed, which needed to be upgraded.)
# --- the Tensorflow LaPlace Image example (Uses PIL, and scipy.misc)
#
# --- Modified: Mar 7, 2017 - by MCL, to just use image file display
# ---
instead of Python Notebooks, IPython, etc.,
# ---
with negative damping and darker image backgrd.
# ---
(Instead of ripples in a pond, we have
# ---
exploding stars ... )
# --- Produces Initial image, 3 intermediated images, and the final image
# as .jpg files. Requires only: tensorflow, numpy, scipy and Pillow
# and Python 2.7.10.
# --- This example taken from Tensorflow Site:
# ---
https://www.tensorflow.org/tutorials/pdes
# --- and provides a nifty example of manipulating n-dimensional tensors.
# ---
# --- For Python newbies (me!): 1) invoke Python in
terminal shell
# --- 2) >>> execfile("LaplaceMCL.py")
# --- focus on understanding exactly how Tensorflow is reshaping tensors
# ------------------------------------------------------------------------------------------
# --- Import libraries
for simulation
import tensorflow as tf
import numpy as np
import scipy.misc
# --- Import for visualization (works now - jpeg encoder recognized
# import PIL.Image
from PIL import Image, ImageDraw
from io import BytesIO
# xxx from IPython.display import clear_output, Image, display
# --- function for
displaying state of the pond's surface as an image
def DisplayArray(a, fmt='jpeg', rng=[0,1]):
"""Display an array as a picture. """
a = (a - rng[0])/float(rng[1]
- rng[0])*37
a = np.uint8(np.clip(a, 0, 255))
f = BytesIO()
# PIL.Image.fromarray(a).save(f,fmt)
# clear_output(wait = True)
# display(Image(data=f.getvalue()))
# --- write the image
scipy.misc.imsave("tensor.jpg", a)
image = Image.open("tensor.jpg")
image.show()
# --- make interactive session for messing around - can use regular session also
sess = tf.InteractiveSession()
# sess = tf.Session()
# --- computational functions go here...
def make_kernel(a):
"""Transform a 2D array into a convolutional kernel """
a = np.asarray(a)
a = a.reshape(list(a.shape) + [1,1])
return tf.constant(a, dtype=1)
def simple_conv(x,
k):
""" A simplified 2D convolutional operation """
x = tf.expand_dims(tf.expand_dims(x, 0), -1)
y = tf.nn.depthwise_conv2d(x, k,
[1, 1, 1, 1], padding='SAME')
return y[0, :, :, 0]
def laplace(x):
"""Compute the 2D laplacian of an array """
laplace_k = make_kernel([[0.5, 1.0, 0.5],
[1.0, -6., 1.0],
[0.5, 1.0, 0.5]])
return simple_conv(x, laplace_k)
# --- Define the PDE - the pond is a perfect 500x500 square
N = 500
# --- now,
we create some raindrops
# --- Initial Conditions -- some rain drops hit the pond
# --- set everything to zero
u_init = np.zeros([N, N], dtype=np.float32)
ut_init
= np.zeros([N, N], dtype=np.float32)
# Some raindrops hit pond at random points (or stars in a starfield)
for n in range(40):
a,b = np.random.randint(0, N, 2)
u_init[a,b] = np.random.uniform()
# --- Create and Display the jpeg image...
DisplayArray(u_init, rng=[-0.1, 0.1])
# Parameters
# eps -- time resolution
# damping -- wave damping
eps = tf.placeholder(tf.float32, shape=())
damping = tf.placeholder(tf.float32, shape=())
# --- Create
vaiables for simulation state
U = tf.Variable(u_init)
Ut = tf.Variable(u_init)
# --- Discretized PDE update rules
U_ = U + eps * Ut
Ut_ = Ut + eps * (laplace(U) - damping * Ut)
# --- Operation to update the state
step = tf.group(
U.assign(U_),
Ut.assign(Ut_))
# --- Run the simulation forward with a simple FOR loop.
# --- Initialize sate to initial conditions
tf.global_variables_initializer().run()
# --- Run
1700 steps of PDE
for i in range(1700):
# Step simulation (damping was 0.04, I made it negative .14)
step.run({eps: 0.03, damping: -0.14})
if (i == 400) : DisplayArray(U.eval(), rng=[-0.1, 0.1])
if (i == 1000) : DisplayArray(U.eval(), rng=[-0.1, 0.1])
if (i == 1400) : DisplayArray(U.eval(), rng=[-0.1, 0.1])
# --- Ok, we are done...
DisplayArray(U.eval(), rng=[-0.1, 0.1])
# --- show the jpeg. Not needed if using DisplayArray.
# image = Image.open("tensor.jpg")
# image.show()
Questions to gemesyscanada at gmail dot com, if curious.
sAPL code:
I've imaged some of the sAPL page-frames, to show terse, useful sAPL code (reads in a text file, processes it into 32-bit floats, plus some binary-ascii conversion examples. One can read the Xerion network weights file into sAPL, and use tanh and dot-product multiply to activate (ie. "run") the Xerion-trained neural network. See images for sAPL code below]I
The sAPL interpreter, as an .exe, is available on the GEMESYS Github account. It is a full APL interpreter, it has no in-app adverts, and is free, and may be copied freely. It is the original IP Sharp APL, running in an emulator for IBM 370 assembler. It runs in a DOSbox, itself also an emulator, but this means sAPL can be run on Windows under cmd shell, on Linux using DOSemu, on an iPad using DOSpad, and it is available at no charge, (no tracking or in-app adverts) from Google PlayStore (search for "GEMESYS" and select "sAPL" and download to any Android device.)
The examples here show how it can be used to "run" the network which has been trained using Xerion, by loading in the high-precision (17 digits real) ascii-text network weights file, and using those weights to process a boolean training case. The sAPL interpreter, which is running on an "jailbroken" iPad, (which is running gDOSpad - the custom version of DOSbox for the iPad) is used to do this. The example screenshots show the APL functions that do this. In this way, the iPad can be used as a portable AI executor, to examine real-time boolean jump-delta vectors (driven by the market prices), and provide predictive information that is actionable.
Some examples of translate functions, overlaid on image of our homeworld, imaged from the International Space Station. The binary table bottom-right, is courtesy of xkcd site. I learned APL first, and I tend to use it to pseudo-code a problem. But once pseudo-coded, it can usually just be transcribed into a runable program. When machines were slow, performance was an issue. Now, performance (execution-speed) is not an issue, typically. It is the human-performance that is causing issues now (pretty much as it always has been, throughout all of history.)
Proof-of-Concept Prototype: (or "The Money Shot"!) This is Actnet2, running against a full boolean jump-delta vector case (created by MAKECASE), and generating the same results as the Xerion-trained neural network. Wtab1 and Wtab2 are the network weights data. Btab are the node bias values. Function Actnet2 runs the Xerion-trained network, for the two boolean "Example Case" vectors, and generates and displays network output. This demonstrates the feasibility of the "Helper AI" concept, using existing technology. Market price-changes, beyond a certain level, can be subjected to a quick evaluation by a tablet device, and action can be taken, if appropriate. The AI can offer immediate, actionable advice, the accuracy of which can be subject to formal evaluation. That way, you can determine if it is actually providing a real edge.
Montage image comparison of two different networks, (left window is newest network, right window is original network). The two networks were trained on same dataset, and are here shown run against the small evaluation training cases (shown in middle screen). The two different neural networks are reporting different results, which suggests that training down to a low sum-squared error level does not necessarily make for predictability. Original network seems to offer a more accurate forecasting outcome. This shows example of why training to lowest sum-squared error is not necessarily as good as training to other objectives, such as maximized cross-entropy.