diff --git a/Examples/README.md b/Examples/README.md index 97a2468..4463edb 100644 --- a/Examples/README.md +++ b/Examples/README.md @@ -6,6 +6,7 @@ * [Web Server](#web_server.js) * [Word Counting](#Word\ Count) * [Implied EMS Operations](#harmony_proxies.js) +* [Inter-language Programming](#Inter-language\ Programming) ## Simple Loop Benchmarks The original [STREAMS](https://www.cs.virginia.edu/stream/) @@ -193,9 +194,9 @@ documents from Project Gutenberg. ## harmony_proxies.js Ordinary JS objects can be made into EMS objects by wrapping them using ES6 proxies. Access to the object uses EMS `read` and `write` operations - that do not use full/empty tag bits preserving legacy load/store semantics, - but inherit the atomic and transactional capabilities of EMS. - The syntax allows incrementally adding EMS operations to legacy programs. +that do not use full/empty tag bits preserving legacy load/store semantics, +but inherit the atomic and transactional capabilities of EMS. +The syntax allows incrementally adding EMS operations to legacy programs. ```javascript emsData["foo"] = 123 // Equivalent to emsData.write("foo", 123) @@ -205,3 +206,26 @@ emsData.readFE("foo") // 123, read "foo" when full and atomically mark empty emsData.writeEF("foo", "one two three") // Write "foo" when empty and mark full emsData["foo"] // "one two three", Full/empty tags not used ``` + + +## Inter-language Programming + +The programs `interlanguage.js` and `interlanguage.py` demonstrate sharing +objects between Javascript and Python. +A variety of synchronization and execution models are shown. + +### Possible Orders for Starting Processes +| Persistent EMS Array File | (Re-)Initialize | Use | +| :------------- |:-------------:| :-----:| +| Already exists | A one-time initialization process truncates and creates the EMS array. The first program to start must create the EMS file with the ```useExisting : false``` attribute | Any number of processes may attach to the EMS array in any order using the attribute `useExisting : true` | +| Does not yet exist | Subsequent programs attach to the new EMS file with the `useExisting : true` attribute | N/A | + + +### Synchronizing Processes +EMS collective operations like barriers and parallel loops are +not available in `user` parallelism modes like inter-language programs. +An example of a simple two process barrier is shown in the example, +a more scalable and robust implementation can be found in the EMS +source code. + + diff --git a/Examples/interlanguage.js b/Examples/interlanguage.js new file mode 100644 index 0000000..2b8bea9 --- /dev/null +++ b/Examples/interlanguage.js @@ -0,0 +1,209 @@ +/*-----------------------------------------------------------------------------+ + | Extended Memory Semantics (EMS) Version 1.4.4 | + | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com | + +-----------------------------------------------------------------------------+ + | Copyright (c) 2017, Jace A Mogill. All rights reserved. | + | | + | Redistribution and use in source and binary forms, with or without | + | modification, are permitted provided that the following conditions are met: | + | * Redistributions of source code must retain the above copyright | + | notice, this list of conditions and the following disclaimer. | + | * Redistributions in binary form must reproduce the above copyright | + | notice, this list of conditions and the following disclaimer in the | + | documentation and/or other materials provided with the distribution. | + | * Neither the name of the Synthetic Semantics nor the names of its | + | contributors may be used to endorse or promote products derived | + | from this software without specific prior written permission. | + | | + | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | + | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | + | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | + | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC | + | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, | + | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, | + | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR | + | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF | + | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING | + | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | + | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | + | | + +-----------------------------------------------------------------------------*/ +/* + Start this JS program first, then start the Python program + + Possible Orders for Starting Processes + --------------------------------------------- + + A persistent EMS file already exists + 1. A one-time initialization process creates the EMS array + 2. Any number of processes my attach to the EMS array in + any order. + + + A new EMS array will be created + 1. The first program to start must create the EMS file with + the "useExisting : false" attribute + 2. Subsequent programs attach to the new EMS file with the + "useExisting : true" attribute + + */ +"use strict"; +let ems = require("ems")(1, false, "user"); // User mode parallelism -- no EMS parallel intrinsics + +const maxNKeys = 100; +const bytesPerEntry = 100; // Bytes of storage per key, used for key (dictionary word) itself +let shared = ems.new({ + "ES6proxies": true, // Enable native JS object-like syntax + "useExisting": false, // Create a new EMS memory area, do not use existing one if present + "filename": "interlanguage.ems", // Persistent EMS array's filename + "dimensions": maxNKeys, // Maximum # of different keys the array can store + "heapSize": maxNKeys * bytesPerEntry, + "setFEtags": "empty", // Set default full/empty state of EMS memory to empty + "doSetFEtags": true, + "useMap": true // Use a key-index mapping, not integer indexes +}); + + + +//------------------------------------------------------------------------------------------ +// Begin Main Program + +// One-time initialization should be performed before syncing to Py +shared.writeXF("nestedObj", undefined); + +// Initial synchronization with Python +// Write a value to empty memory and mark it full +shared.writeEF("JS hello", "from Javascript"); + +// Wait for Python to start +console.log("Hello " + shared.readFE("Py hello")); + +/* + This synchronous exchange was a "barrier" between the two processes. + + The idiom of exchanging messages by writeEF and readFE constitutes + a synchronization point between two processes. A barrier synchronizing + N tasks would require N² variables, which is a reasonable way to + implement a barrier when N is small. For larger numbers of processes + barriers can be implemented using shared counters and the Fetch-And-Add + (FAA) EMS instruction. + + The initialization of EMS values as "empty" occurs when the EMS array + was created with: + "setFEtags": "empty", + "doSetFEtags": true, + If it becomes necessary to reset a barrier, writeXE() will + unconditionally and immediately write the value and mark it empty. + */ + + +// Convenience function to synchronize with another process +// The Python side reverses the barrier names +function barrier(message) { + console.log("Entering Barrier:", message); + shared.writeEF("py side barrier", undefined); + shared.readFE("js side barrier"); + console.log("Completed Barrier."); + console.log("---------------------------------------------------"); +} + + +// -------------------------------------------------------------------- +barrier("Trying out the barrier utility function"); +// -------------------------------------------------------------------- + + +// JS and Python EMS can read sub-objects with the same syntax as native objects, +// but writes are limited to only top level attributes. This corresponds to the +// implied "emsArray.read(key).subobj" performed +shared.top = {}; +console.log("Assignment to top-level attributes works normally:", shared.top); +shared.top.subobj = 1; // Does not take effect like top-level attributes +console.log("But sub-object attributes do not take effect. No foo?", shared.top.subobj); + +// A nested object can be written at the top level. +shared.nestedObj = {"one":1, "subobj":{"left":"l", "right":"r"}}; +console.log("Nested object read references work normally", shared.nestedObj.subobj.left); + +// A workaround to operating on nested objects +let nestedObjTemp = shared.nestedObj; +nestedObjTemp.subobj.middle = "m"; +shared.nestedObj = nestedObjTemp; + +// The explicitly parallel-safe way to modify objects is similar to that "workaround" +nestedObjTemp = shared.readFE("nestedObj"); +nestedObjTemp.subobj.front = "f"; +shared.writeEF("nestedObj", nestedObjTemp); + + +// -------------------------------------------------------------------- +barrier("This barrier matches where Python is waiting to use nestedObj"); +// -------------------------------------------------------------------- + +// Python does some work now +// Initialize the counter for the next section +shared.writeXF("counter", 0); + +// -------------------------------------------------------------------- +barrier("JS and Py are synchronized at the end of working with nestedObj"); +// -------------------------------------------------------------------- + +// --------------------------------------------------------------------------- +// Use the EMS atomic operation intrinsics to coordinate access to data +// This is 10x as many iterations as Python in the same amount of time +// because V8's TurboFan compiler kicks in. +for (let count = 0; count < 1000000; count += 1) { + let value = shared.faa("counter", 1); + if (count % 100000 == 0) { + console.log("JS iteration", count, " Shared Counter=", value) + } +} + + +// -------------------------------------------------------------------- +barrier("Waiting for Py to finish it's counter loop"); +// -------------------------------------------------------------------- + +console.log("The shared counter should be 11000000 ==", shared.counter); + + + +// --------------------------------------------------------------------------- +// Ready to earn your Pro Card? + +// Wait for Py to initialize the array+string +barrier("Wait until it's time to glimpse into the future of Javascript"); +console.log("The array+string from Py:", shared.arrayPlusString); + +barrier("Waiting for shared.arrayPlusString to be initialized"); +console.log("JS can do array+string, it produces a string:", shared.arrayPlusString + " world!"); +shared.arrayPlusString += " RMW world!"; +console.log("JS performing \"array + string\" produces a self-consistent string:", shared.arrayPlusString); +barrier("Arrive at the future."); + + +// --------------------------------------------------------------------------- +// Leave a counter running on a timer +// First re-initialize the counter to 0 +shared.writeXF("counter", 0); + +function incrementCounter() { + // Atomically increment the shared counter + let oldValue = shared.faa("counter", 1); + + // If the counter has been set to "Stop", clear the timer interval + if (oldValue == "stop") { + console.log("Timer counter was signaled to stop"); + clearInterval(timerCounter); + } + + // Show counter periodically + if (oldValue % 100 == 0) { + console.log("Counter =", shared.counter); + } +} + +var timerCounter = setInterval(incrementCounter, 10); // Increment the counter every 1ms + +barrier("Welcome to The Future™!"); +// Fall into Node.js event loop with Interval Timer running diff --git a/Examples/interlanguage.py b/Examples/interlanguage.py new file mode 100755 index 0000000..8467e0b --- /dev/null +++ b/Examples/interlanguage.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +""" + +-----------------------------------------------------------------------------+ + | Extended Memory Semantics (EMS) Version 1.4.4 | + | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com | + +-----------------------------------------------------------------------------+ + | Copyright (c) 2017, Jace A Mogill. All rights reserved. | + | | + | Redistribution and use in source and binary forms, with or without | + | modification, are permitted provided that the following conditions are met: | + | * Redistributions of source code must retain the above copyright | + | notice, this list of conditions and the following disclaimer. | + | * Redistributions in binary form must reproduce the above copyright | + | notice, this list of conditions and the following disclaimer in the | + | documentation and/or other materials provided with the distribution. | + | * Neither the name of the Synthetic Semantics nor the names of its | + | contributors may be used to endorse or promote products derived | + | from this software without specific prior written permission. | + | | + | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | + | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | + | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | + | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC | + | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, | + | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, | + | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR | + | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF | + | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING | + | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | + | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | + | | + +-----------------------------------------------------------------------------+ +""" +""" + tl;dr Start the Javascript first, then the Python + -------------------------------------------------- + + See interlanguage.js for more detailed description of starting processes + sharing EMS memory + + NOTE ABOUT "STUCK" JOBS": + Python runtime can get "stuck" and may not respond to Control-C + but can still be interrupted with Control-Z or Control-\ + This "stuck" behavior is due to how the Python/C interface protects + the Python runtime. +""" + +import time +import sys +import random +sys.path.append("../Python/") +import ems + +# Initialize EMS: 1 process, no thread-core affinity, user provided parallelism +ems.initialize(1, False, "user") + + +# The EMS array attributes must be the same by all programs sharing the array, +# with the exception of the the initial file creation which uses "useExisting:False" +# instead of "useExisting:True" +maxNKeys = 100 +bytesPerEntry = 100 # Bytes of storage per key, used for key (dictionary word) itself +shared = ems.new({ + 'useExisting': True, # Connect the EMS memory created by JS, do not create a new memory + 'filename': "interlanguage.ems", # Persistent EMS array's filename + 'dimensions': maxNKeys, # Maximum # of different keys the array can store + 'heapSize': maxNKeys * 100, # 100 bytes of storage per key, used for key (dictionary word) itself + 'useMap': True # Use a key-index mapping, not integer indexes +}) + + +# ------------------------------------------------------------------------------------------ +# Begin Main Program +print """Start this program after the JS program. +If this Python appears hung and does not respond to ^C, also try ^\ and ^Z +---------------------------------------------------------------------------- +""" + +# Initial synchronization with JS +# Write a value to empty memory and mark it full +shared.writeEF("Py hello", "from Python") + +# Wait for JS to start +print "Hello ", shared.readFE("JS hello") + + +def barrier(message): + """Define a convenience function to synchronize with another process + The JS side reverses the barrier names""" + global shared + naptime = random.random() + 0.5 + time.sleep(naptime) # Delay 0.5-1.5sec to make synchronization apparent + print "Entering Barrier:", message + shared.writeEF("js side barrier", None) + shared.readFE("py side barrier") + print "Completed Barrier after delaying for", naptime, "seconds" + print "------------------------------------------------------------------" + + +print "Trying out the new barrier by first napping for 1 second..." +time.sleep(1) +# --------------------------------------------------------------------- +barrier("Trying out the barrier utility function") +# --------------------------------------------------------------------- + + +# -------------------------------------------------------------------- +# This barrier matches where JS has set up "shared.top", and is now waiting +# for Python to finish at the next barrier +barrier("Finished waiting for JS to finish initializing nestedObj") + +print "Value of shared.nestedObj left by JS:", shared.nestedObj +try: + shared.nestedObj.number = 987 # Like JS, setting attributes of sub-objects will not work + print "ERROR -- This should not work" + exit() +except: + print "Setting attributes of sub-objects via native syntax will not work" + +print "These two expressions are the same:", \ + shared["nestedObj"].read()["subobj"]["middle"], "or", \ + shared.read("nestedObj")["subobj"]["middle"] + + +# -------------------------------------------------------------------- +barrier("JS and Py are synchronized at the end of working with nestedObj") +# -------------------------------------------------------------------- + + +# Enter the shared counter loop +for count in xrange(100000): + value = shared.faa("counter", 1) + if count % 10000 == 0: + print "Py iteration", count, " Shared counter=", value +barrier("Waiting for Js to finish it's counter loop") + + +# -------------------------------------------------------------------- +# Ready to earn your Pro Card? + +# Make JS wait while Python demonstrates a "create approach to language design" +shared.arrayPlusString = [1, 2, 3] # Initialize arrayPlusString before starting next phase +barrier("Wait until it's time to glimpse into the future of Python") +try: + dummy = shared.arrayPlusString + "hello" + print "ERROR -- Should result in: TypeError: can only concatenate list (not \"str\") to list" + exit() +except: + print "Adding an array and string is an error in Python" + +shared.arrayPlusString += "hello" +print "However 'array += string' produces an array", shared.arrayPlusString + +# Let JS know 'shared.arrayPlusString' was initialized +barrier("shared.arrayPlusString is now initialized") +# JS does it's work now, then wait for JS to do it's work +barrier("Arrive at the future.") +# JS does all the work in this section +barrier("Welcome to The Future™!") + + +# -------------------------------------------------------------------- +# JS is now in it's event loop with the counter running +for sampleN in xrange(10): + time.sleep(random.random()) + print "Shared counter is now", shared.counter + +# Restart the counter +shared.counter = 0 +for sampleN in xrange(5): + time.sleep(random.random() + 0.5) + print "Shared reset counter is now", shared.counter + + +# Wait 1 second and stop the JS timer counter +time.sleep(1) +shared.counter = "stop" + +# Show the counter has stopped. Last value = "stop" + 1 +for sampleN in xrange(5): + time.sleep(random.random() + 0.5) + print "Shared reset counter should have stopped changing:", shared.counter + +print "Exiting normally." diff --git a/README.md b/README.md index dd37f8b..fc8c858 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,9 @@ without using distributed programming. ## Sharing Persistent Objects Between Python and Javascript + +Inter-language example in [interlanguage.{js,py}](https://github.com/SyntheticSemantics/ems/tree/master/Examples) + * Start Node.js REPL, create an EMS memory * Store "Hello" * Open a second session, begin the Python REPL diff --git a/package.json b/package.json index 3847af1..6425e56 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "ems", - "version": "1.4.3", + "version": "1.4.4", "author": "Synthetic Semantics ", "description": "Persistent Shared Memory and Parallel Programming Model", "contributors": [