In a View of a Software Project as a Tree-Like Structure: Towards Memory Locking and Memory Access Control in Common Lisp Programs

On beginning a study of National Instruments' LabVIEW programming platform, last year -- that, as was contingent on being a student of a sort of vocationally-focused introductory program in computing, networking, and the electrical sciences, a program hosted by a certain online university -- the author if this article has begun studying a broader concept of data flow programming. Contingent with that study -- not as if try to trump-up any competition towards National Instruments (NI) but rather towards a broader academic study of data flow programming -- the author has been developing an expression of a concept of data flow programming, in Common Lisp.

The author would gladly make reference to the source-tree for that item, here, but it may not be in any useful state for applications, presently. The author has been referring to the project as eSym -- as to denote something towards a concept combining of component concept of symbolic logic and a metaphor with regards to electrical current flow.


While developing eSym, the author has discovered a number of complimentary concerns that may be addressed in a Common Lisp program. Those concerns may be addressed before eSym would be any further developed.

The following article presents an outline of those initial concerns, then proceeds to develop some concepts as with regards to a sense of a broader context in development software programs in Common Lisp.

• An extension may be defined onto the Common Lisp Object System (CLOS) -- at the least, in implementations applying of the Metaobject Protocol (MOP) -- such that the extension would allow for a thread-local modal locking procedure -- and more specifically, a read/write locking procedure -- onto slot values of a Common Lisp standard object.
• This, in turn, may be implemented onto a portable interface for modal locking, such that may be developed directly onto Bordeaux Threads (BT), in a platform-agnostic model
• In parallel to developing a modal locking layer onto Bordeaux Threads, an additional portability interface may be developed onto so many implementation-specific timer interfaces
• ....such that may then be applied in a revised `BT:WITH-LOCK` then allowing a blocking keyword argument -- the behaviors of which would be fully described in the documentation, but in summary of which: 
• blocking t : block indefinitely to acquire lock
• blocking nil : do not block. Rather than returning nil, should signal error of type lock-held if lock is held, such that the calling function could then handle appropriately without storing a return-value from the failed lock acquisition request
• blocking {integer} : block for a duration specified by {integer}, and if timeout, then signal error of type timeout-exceeded
• ....and that may then also be applied in a new `WITH-MODAL-LOCK` macro, as well as underlying `ACQUIRE-MODAL-LOCK` function
• Alternate to Bordeaux Threads:
• Threading in cl-async [github]
• Extended with a BT-like interface in Green Threads
• CPS: Continuation Passing Style [multscheme]
• In operating systems applying a pthreads threading model, this may provide an interface onto an underlying PThread implementations, as via cl-libevent2 and transitively, libevent
• libevent provides a platform-agnostic model for development of asynchronous programs onto C
• As far as a portable timers implementation in recent revisions of cl-async : See also  delay / with-delay (?)
• The data flow programming language, in its abstract graphical representation, could be implemented as an extension of the abstract graphical syntax and the underlying semantics of the Systems Modeling Language (SysML) ... as SysML being implemented with a metamodel extending of the meta-metamodel defined of the Metobject Framework (MOF).
• Alternately, a SysML view for the data flow programming language's data flow graphs could be added later.
• This may extend, indirectly, of de.setf.xml -- with de.setf.xml providing an interface for parsing the XMI schema and the XMI schema then serving to present a structured interface for processing and I/O onto metamodels serialized onto the standard MOF/XMI binding , with any number of vendor-specific conventions in the same.
• An implementation of the Object Constraint Language (OCL) should be developed as part of the same -- such that may apply the ATN parser, such that is applied in de.setf.xml -- the parser implementation in the OCL implementation therefore remaining aligned with the XML implementation
• This data flow programming model may  of course be implemented with extensions onto McCLIM, for managing interactions with a data flow programming graph, as via a CLIM application frame.

In parallel to the matter of thread-local slot value locking: While I was developing some notes as with regards to a lock ticket caching framework -- and a more generic object buffering framework -- as to address a question of how a program may provide an interface for controlling access to the slots of  buffered object, such that the buffered objectwould be simultaneously accessible in a buffer cache and from there, accessible in any number of threads of a program, as well as being accessible broadly in the data space of the containing process -- I had developed a question as to how and whether it may be possible to control access to a region of memory. In regards to how that may be approached in software programs developed on the Linux kernel, my question -- as such -- was promptly answered, on discovering the shmget(2) and shmctl(2) manual pages, as well as the useful  shm_overview(7) such that provides an overview of an independent SHM interface providing file descriptors for shared memory segments.  I would denote some additional caveats to this matter, as well:

• The  shmctl(2) manual page, specifically, describes some of the structural qualities of the following structure types in the Linux programming environment:
• The C structure type shmid_ds 
• Application: This type appears to be used for providing information about the Kernel's implementation of the SHM interface. For instance, there are some structure slots for recording of times and process IDs (PIDs), within the shmid_ds structure type
• The shm_perm slot provides further structure, as denote in the following
• The C structure type ipc_perm
• This type uses the key value as applied also with shmget(2)
• This type also provides a record for UID and GID values as well as flags for access permissions defined onto a shared memory segment
• Consequent to a further study of the shmctl(2) manual page, it may seem that the shmctl() function is defined for purpose of reflection onto system-wide registers with regards to shared memory allocation
• An orthogonal question: Onto what platforms other than Linux is the Linux SHM interface considered portable?
• BSD?
• QNX?
• It's not the same as mlock(2)
• Application: Ideally, there may be a compatibility interface defined between values used in  shmget(2) and similar SHM functions, and values used in mlock(2). Such a compatibility interface may or may not exist, presently, within the body of existing work in free/open source software and systems
• mlock(2) may be applied in at least two manners of usage case
• To prevent a memory segment from being paged to disk, as to ensure that data in the memory segment will not be recorded in any permanent medium, namely towards applications in data security
• To ensure that a segment of memory will not be paged to disk, as towards ensuring a deterministic timing within the calling program, for access onto the same segment of memory -- as towards applications in a realtime operating system (RTOS) environment, such as for embedded computing. As a further reference, to that effect: sched_setscheduler(2)

This outline, of course, does not present a full interface onto SHM in Linux, for any single Common Lisp implementation. The previous outline may serve to develop some further knowledge, towards some questions with regards to memory management, within any single Common Lisp implementation, viz a viz:

• One  "first question", as towards the origins the previous outline -- more broadly, as towards preventing unwanted access to regions in a program's data space -- in that regards, towards a design for data security within Common Lisp programs. 
• The question, in short: How can an interface be developed for controlling access to memory registers within a program's data space, in a Common Lisp program on any single operating system platform? 
• Nothing to compete with the Java programming model, certainly, though Java also defines some controls for data access, at least insofar as with regards to definitions of Java fields and Java methods in Java classes. The author is not presently aware of any similar SHM interfaces in Java.
• A second question, as contingent around the matter of RTOS applications: 
• The question, in short: How can memory be locked, within a Common Lisp program -- whether permanently or temporarily -- as to not be shifted in the data space, within garbage collection processes?
• This is orthogonal to the question: How can memory be locked, within a Common Lisp program, as to not be paged out to the system's page area?
• The "second question" is notably a naive question, it demonstrating some assumptions as to what side effects a garbage collection (GC) process may produce, within a Common Lisp program. Namely, there is an assumption that an object's object address may change within the duration of a Common Lisp program -- assuming that an object in a Common Lisp program may not be left where originally created within a program's data space, after completion of any single GC iteration.
• Of course there would be some orthogonal concerns, and perhaps some orthogonal design concepts, for instance:
• Is it possible to define a specific region of memory as being locked with mlock(2), and such that that region of memory may be used for allocating multiple Lisp objects? 
• May that serve to provide a methodology for implementing a non-paged memory IO region wtihin a Common Lisp program?
• May it serve to allow for a tunable efficiency in applications, such that mlock(2) may not need to be applied at every time when a new Lisp object is allocated? 
• If a shared memory control interface would implemented, in parallel to this feature, of course the shared memory control interface may produce some side effects as with regards to this feature's implementation -- for instance, in that an access-controlled region of memory may be effectively contained within the mlock'd memory region.
• There may be some additional concerns addressed, as with regards to manual memory management in Common Lisp programs, for instance:
•  That  if there may be an application in which one cannot simply cons and forget and leave the garbage collector to clean up the memory space for unused objects
• ...then as well as the implicit memory allocation of creating a new Lisp object
• ...a Lisp program may provide an interface for manual memory deallocation, thus towards both
• obviating the burdens for the garbage collector and also 
• allowing for a deterministic procedure chain, insofar as for memory management within a Common Lisp program
• This can be implemented with requirement for multithreading.
• An explicit memory deallocation thread may be implemented
• ...as to be coordinated in scheduling alongisde the garbage collector
• ...such that it would be accompanied with rigorous definitions of memory deallocation forms, for ensuring that objects created for "one time use" within a single lexical environment will be explicitly marked as to deallocate when control exits from within the same lexical environment.
• ...though that would require a storage of a separate "to-deallocate table" essentially separate to reference-counted memory -- lazy references? -- and a methodology for iterating across the same table, perhaps alternate to a methodology of scanning Lisp objects recursively for count of object references and recursively deallocating those denoted only as "not referenced".
• ...and may be applied with an functional interface, parallel to cl:gc, rather such as foo-mem:prune
• ...such that an application applying the memory prune function may be exist in parallel to applications not doing such explicit memory deallocation.
• ...and should be accompanied with a conditions model, such as for when   foo-mem:prune would be applied on any object that would contain, as an n-ary  referenced object, any object that would be known as that it cannot be pruned, for instance any definition of a class defined in the cl package
• ...and must also be accompanied with a lengthy description of how foo-mem:prune would function on individual objects to be pruned
• Note: Reference vs definition -- for instance, that a reference to a string is not a string object itself
• Essentially, this may serve to develop a concept: Memory management for objects referenced within only a single lexical environment
• Ideally, this should be implemented as to not entail any overhead for "Reference counting".

Certainly, a Common Lisp interface on the SHM features of the Linux kernel may be implemented as a layer onto an interface onto mlock(2) -- and so, the development of the mlock(2) interface could effectively precede the development of the SHM memory control interface. Also clearly, such extrnsions would need to be implemented in all of an implementation-specific manner, as onto the memory management features of any single Common Lisp implementation. This, then, would certainly require much of a detailed study onto those same features, and perhaps an application of the C programming language.

Some further notes in documentation should be forthcoming, to that effect, as may be published in this web log, henceforward. The author proposes to limit the study onto two Common Lisp implementations specifically:

• Steel Bank Common Lisp (SBCL)
• Clozure Common Lisp (CCL)

...at such future time as when it may be immediately feasible as to address the following procedural outline into an implementation:

1. To begin such a study of memory management, on those platforms.
• Ostensibly, to develop a series of SysML models for describing the conventions applied in memory management on those platforms
• UML class diagrams for illustrating structures and relations of explicit and inferred data types (e.g. VOPs in CMUCL and SBCL)
• SysML block diagrams for developing a block/port-oriented view of memory and memory management  in the respective Common Lisp implementation
• Onto CCL, block diagrams illustrating of qualities of the FFI interface
• State Charts in SysML for describing memory management in terms of finite state automota
2. To develop an interface for an  mlock(2) locked region of memory in a program's data space
3. Implicitly: To develop a Common Lisp interface onto an operating system's underlying authentication and authorization (A2) model
• To furthermore extend that interface onto the A2 procedures of Kerberos 
4. To develop an interface for an SHM controlled access to memory within a program's data sapce
• To develop such condition types, deubugger features, and other features unto the Common Lisp semantics, such that may serve to simply the programmer's view of the memory access control forms 

If the reader may wish to inquire as to why those two platforms are specified, in this article:

• Both of those platforms may be applied on an ARM microprocessor architecture
• ARM, in turn, is applied within a number of embedded computing platforms, including mobile phones
• Although there are additional Common Lisp implementations available for the ARM architecture -- including ECL -- it may be feasible to limit the study to, essentially: 
• One implementation of or deriving from CMUCL, as with SBCL at the original "Point of fork"
• This should be approached with an observation about signals, interrupts, and foreign threads in SBCL -- as denoted as being problematic for an interface onto a Java Virtual Machine, at the web site for CL+J 
• This should be approached, also, with an observation as with regards to the newer thruption model in SBCL, and related features defined in recent editions of SBCL
• One implementation deriving its implementation more broadly from LibC, as with CCL and its own respective foreign functions interface
• This may be approached with consideration: 
• That the Linux Kernel is implemented in C
• That the fundamental interfaces to the Linux Kernel are provided as for C programs
• That the FFI model developed in CCL may applied in a prototype fork for and towards applying such extended memory management procedures as denoted in the previous outlines in this article

Of course, it would be impractical to assume as if an extension could be addressed instantaneously onto two separate Common Lisp implementations, if only a single developer is developing the same extension. At this point in the article, the author arrives at a point of decision as to which Common Lisp implementation the author would wish to focus on, first, if to develop such an extension. That question, then, may be pushed further down the effective stack, with a couple of corresponding questions:

• How may a mobile application development model be developed in free/open source software, for supporting development of mobile applications and desktop applications with Common Lisp?
• How may CLIM be applied within such a model for mobile application development in a free/open source regards?
• How may that be extended, furthermore, in applications of "new" embedded computing models, such as may apply the BeagleBone Black platform, Gumstix, or any other single-board computing model capable of running a Linux operating system?
• How easily can CCL or SBCL and McCLIM be installed onto an Android device, for purpose of development testing?
• How easy would it be to use the CLIM REPL, in such an installation?
• How many separate concerns may be addressed onto CLIM, as for design of mobile human computer interfaces -- in whatever regards, if as an alternative to conventions developed of popular desktop computing interfaces? 
• How and how well may CLIM be integrated with an on-screen keyboard, as of a tablet PC or a notebook with touschscreen?
• Would it not be useful to include a modeline -- even if it would be a 'hidden modeline' of a kind -- onto every single window created, in such an appliation of CLIM? Then how would the 'meta' key be implemented, for key input to such a modeline?
• How can a commercial interface be developed, onto such work, and it not be limiting towards the free open source nature of the software that would be applied in such work?
• Sidebar: "Free/open source" does not imply "Insecure"
• A deterministic model will not be illustrated for that thesis, in this article
• Sidebar: Incentives?
• Data-secure computing for small/medium enterprise - referencing strongSwan, Kerberos, and JacORB as providing support for Kerberos and SSL onto CORBA
• Games programming for mobile platforms, in Common Lisp? 
• Can we make it look more like Half Life? Should we?
• What about music?
• What about folk music?
• What about educational applications developed albeit with a novel game-like interface but lending towards any actual, fundamental concepts in mathematics, if not also towards the material sciences?
• What wonders may await, in the literary works of Charles Babbage?
• What wonders, in a view providing of an implicit sense of gender neutrality, if not a broader sense of cultural neutrality, with regards to mathematics and science? barring any discussions lending immediately to a maturity of nature.
• Furtherance of concepts in the state of the art?
• IDE development in the heterogenous systems programming environment
• Towards implications for the software development environments in applications of virtualizaiton and software-defined networking (SDN)
• "Because CORBA"
• Beginnings of discussion with regards to community support, if not for further commercial support for Common Lisp  applications on mobile platforms?
•  "There's the kicker"
• Existing services for issue support may include:
• YouTrack
• Bugzilla
• Request Tracker
• GNATS
• Debian BTS
• Bug tracking in Canonical's Launchpad system
• May entail some platform-specific focus about Linux
• The following mobile platforms implement Linux:
• Maemo
• Android
• Fire OS (Kindle Fire)
• Firefox OS
• Chromium OS
• Any single single-board computing platform on which Linux is available
• BeagleBone Black (BBB)
• Can run Android, more specifically. See also: Rowboat
• Raspberry Pi (RPi)
• Gumstix
• Etc.
• RTOS features "To be determined" by platform (TBD)
• Focusing on the Android platform, specifically, there may be an interface available for bug tracking onto any single Anrdoid app store (TBD)
• Questions with regards to tools
• A hypothetical model for capturing results of an application error: Fork, save diagnostic data-optionally save entire Lisp image, and debug on 'Own hardware"
• Revisions for "Other users" : Do not save session-local data within diagnostics; Do not save Lisp image; ensure full authorization and authentication with upstream issue tracking agency, in network transactions transmitting of diagnostic information
• Debugger hook : How to apply that usefully on a mobile platform?
• Usage Case - Developer's mobile platform
• Context: Prototyping and testing
• Context: Unexpected situation in program
• Usage Case - User's mobile platform
• Context: Regular application procedures
• Context: Unexpected situation in program
• Debugger hook   How to apply that  usefully, for server diagnostics?
• Data models : How to "flag" session-local data values, so as to prevent their printing or storage onto media outside of environment of a running Common Lisp program?
• Systems models : How to securely and conveniently (?) encrypt a saved Lisp image?
• Further implications onto design and development of digital platform operating systems?
• How to integrate developer tools, in a novel manner, with Xen? 
• cf. Xen Dom0 
• Emulator as Dom1 instance
• Xen must be (?) installed directly on platform hardware
• Orthogonal to: Microkernel design.
• cf. Amazon Web Services
• Application of Xen

To this point in the article , the article has diverged from the original topic of developing a data flow programming model in Common Lisp.  Figuratively, the eSym project might represent some figurative icing on the figurative concha otherwise described in this article. The majority of such tasks as would entail an extension of interfaces onto the Linux kernel -- as addressed in this article, albeit superficially -- those might be approached albeit without a cup full of sugared bread, but with a substantial portion of documentation regardless.

This article has described a substantial number of individual concepts and related task items, such that may be addressed into development of an essentially arbitrary number of source trees.

This is developed under the MetaCommunity project, towards a further goal of developing a comprehensive architecture for application development in Common Lisp, and an orthogonal goal of reviving the design of MIT CADR for application onto contemporary microprocessor architectures, denoted as the CIIDR project.