Wednesday, March 18, 2015

Using Elasticsearch and Moqui for Analytics

Introduction to Elasticsearch in Moqui


Elasticsearch has been used in Moqui for years for search purposes, finding things like tasks, products, and wiki pages. It is a great tool for finding data quickly from a wide variety of sources indexed using Apache Lucene for flexible full-text searching.

The Data Document feature in Moqui makes it easy to define a JSON document (or nested Map/List structure) derived from relational database records. The Moqui Data Feed feature provides a real-time link between Data Documents and tools that use them, including the service built into Moqui that receives documents from a feed to index them for searching and other purposes. This can also be used to send data to other systems, trigger emails, and so on.

Elasticsearch is much more than just a full-text search tool. The faceted searches are helpful when searching structured data, but indexing by different fields within a document can be used for so much more. To get an idea of the scope and usefulness of Elasticsearch look at the home page for it:

https://www.elastic.co/products/elasticsearch

Analytics Capabilities and Easy Use in Moqui


The analytics capabilities of ES combine the distributed big-data and faceted search capabilities with a wide variety of tools for bucketing and aggregating data. For a list of the aggregation functions available, and a summary of the aggregation feature in general, see:

http://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html

It has basic numeric functions for "metrics" like min, max, sum, and avg plus more advanced ones like percentiles, histograms, geographic areas, extracting significant terms, and even scripts for custom metrics. These can be calculated over sets of documents split into "buckets" distinguished by terms, ranges of values, and more. Filters can be applied to the documents included at the top level and in any bucket.

For a more general idea of aggregations they are covered in one part of the book "Elasticsearch: The Definitive Guide" which is available online here:

http://www.elastic.co/guide/en/elasticsearch/guide/current/aggregations.html

For analytics in Moqui this provides an ideal tool for persistence of analytic data derived from operational data. The analytic data can be easily searched with results bucketed and metrics calculated as needed. Feeding data to this analytic data store is easy in Moqui by defining Data Documents that cover entity fields you want to report on, plus a real-time Data Feed to send the documents to Elasticsearch for indexing as records in the operational database are updated.

Real World Examples from Mantle Business Artifacts


For some real world examples of Data Documents useful for analytics see the MantleDocumentInventoryData.xml file in Mantle Business Artifacts:

https://github.com/moqui/mantle/blob/master/mantle-udm/data/MantleDocumentInventoryData.xml

This file has documents for inventory data, both current and projected, based on inventory asset records, sales order items, production estimates, and production run consume/produce estimates and actuals. Over time this will be extended to include shipments, purchase orders, and other inventory-relevant data structures. Most of these documents include data/time fields that can be used to limit projections to a certain period in the future and along with statusId fields exclude data not useful for future projections. Documents from the past can be used for reporting on the history of inventory and statistics about incoming and outgoing items.

A big part of the efficiency in reporting comes from pulling together data from various parts of the operational database into a single document that is persisted and indexed in the analytic data store (in this case Elasticsearch). That is what these Data Document definitions do. They pull together the needed fields from various tables into a single JSON document that can be used for efficient filtered and bucketed searches with metrics calculated on the fly.

Note (if you want to run this stuff) that to actually create and index documents from these Data Document definitions you must uncomment the DataFeed defined at the bottom of this file, and do so before the data load (i.e. 'gradle load').

The service in Moqui to do analytics queries is org.moqui.impl.EntityServices.search#CountBySource which is designed to take a "source" parameter with the details of the search to do in Elasticsearch. Here is an example of a source Map in Groovy syntax:
    andList = [[terms:[productId:productIdList]], [not:[terms:  [partStatusId:   ['OrderCompleted','OrderRejected','OrderCancelled']]]]]
    if (facilityId) andList.add([term:[facilityId:facilityId]])
    if (estThruDateStr) andList.add([or: [[missing:[field:'requiredByDate']], [range:[requiredByDate:[lte:estThruDateStr]]]]])
    searchSourceMap = [
        size: maxResults, query: [filtered: [filter: [and: andList] ]],
        aggregations: [
            products: [
                terms: [field: 'productId', size: maxResults],
                aggregations: [
                    orderQuantitySum: [sum: [field:'orderQuantity']],
                    quantityReservedSum: [sum: [field:'quantityReserved']],
                    quantityNotAvailableSum: [sum: [field:'quantityNotAvailable']],
                    quantityNotIssuedSum: [sum: [field:'quantityNotIssued']]
                ]
            ]
        ]
    ]
This is from the get#InventoryProjectedInfo service in the InventoryReportServices.xml file in Mantle:

https://github.com/moqui/mantle/blob/master/mantle-usl/service/mantle/product/InventoryReportServices.xml

This source Map in Groovy syntax follows the same structure as JSON documents sent to Elasticsearch through its REST API, and is generally easier than the Java API to use plus more closely aligns with the ES documentation examples.

This example is meant to get sums of various values for order items that are associated with order parts that still open and that may or may not be reserved. This is useful to see upcoming orders that will effect inventory at some point in the future.

Pitfalls and Quirks of Elasticsearch for Analytics


One potential issue with Elasticsearch for analytic use is the default result size of 10. When using aggregations with buckets the general "size" parameter passed in to the Java API or REST request parameter does NOT apply to the buckets. If you want more than 10 buckets in an aggregation you must specify a larger size in the bucket's "terms" Map. In the example above the terms Map is [field: 'productId', size: maxResults] using the maxResults field to allow for a higher value (defaults to 1000for the get#InventoryProjectedInfo service).

Another big pitfall is the data type of document fields. If you try to do a sum, avg, etc on a string type field you will get an error, you need a number type field instead. Here is a reference for the core data types available in Elasticsearch:

http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html

The type of each field is specified using a "mapping" for a document type. In the most recent Moqui Framework code it automatically checks to see if an index exists and if not it creates the index and puts all the document type mappings. These type mappings are generated based on the Data Document definition and the entity field type for each field included in it. If you want to see what these mappings look like you can get them from Elasticsearch after the index has been used at least once, to index a document or to search, using a curl command like this:

curl -XGET 'http://localhost:9200/default__mantle_inventory/_mapping?pretty=true'

Note that there is a separate set of indexes for each tenant in Moqui, which is prepended to the index name and separated with a double underscore. The entire extended index name is lower-cased as Elasticsearch only supports lower case index names. This is why there is a "default__" at the beginning of the index name for the DEFAULT tenant in Moqui.

In addition to the numeric field types another important thing Moqui does in the default document mappings is to set all ID fields (entity field types id and id-long) to have index: not_analyzed. This tells Elasticsearch to use and index the literal ID value instead of analyzing it for search terms which also changes the value to all lower case (which is how ES handles case-insensitive searches). Without this fields like productId, partStatusId, facilityId, etc would be lower-cased and possibly split into multiple search terms if they contain certain characters. This seriously messes up both term/terms filters and the ID values that come back for the bucket keys.

There are also some Data Document and Feed quirks in Moqui to be aware of. The main one is that for the real-time push feed (triggered by changes to entity records through the Entity Facade) there must be a reverse-relationship defined for each forward relationship in the field path in a data document field. For example consider this field path in the MantleInventoryOrderItem Data Document:

mantle.order.OrderPart:Vendor#mantle.party.PartyRole:roleTypeId

This uses the relationship "Vendor#mantle.party.PartyRole" from OrderPart to PartyRole. For the real-time push we need the reverse relationship, i.e. one from PartyRole back to OrderPart which can be seen in the "Vendor#mantle.order.OrderPart" relationship on the PartyRole entity.

Conclusion


For more complex analytics, especially that involve data from many places in an operational database with a variety of constraints and with a variety of metrics applied, it is common to have a separate data store that is structured for analytics and the particular ways the data will be used.

Star schemas in relational databases are a nice approach to this, but still limited by the flat relational structure. With hierarchical JSON documents you can include "dimensions" in "fact" documents for more efficient querying/searching while still remaining flexible. You can still have separate documents that are "joined" in a search, but because documents in the database are only loosely typed and structured and because they are hierarchical you are not limited to this.

Relational SQL databases are also generally more difficult to scale for the large amount of data needed when indexing a large number fields across very large numbers of records.

These are some of the reasons that NoSQL and more generally non-relational databases are becoming more popular for analytics. When not so concerned with data consistency and referential integrity, and more generally the ACID properties, sharding and otherwise distributing data across various physical machines is MUCH easier. This is great for reporting in real time on large data sets as the data can be processed on each server and then the aggregated results combined from the various servers to get the final results.

Elasticsearch is a nice tool for this. It features much of the scalability and flexibility of document databases like MongoDB but is WAY easier to deploy in Java applications and usable in so many ways. ES can be run embedded in a Java application, as it is by default in Moqui Framework, or it can be distributed across hundreds of servers coordinating in a cluster with automatic discovery, sharding, and so on. Elasticsearch also does a pretty good job of caching frequent searches, and has all sorts of tools and configuration options for tuning searches, aggregations, and caching.

Elasticsearch also goes beyond what most document databases provide in terms of the variety of aggregations it can run on the document data. This provides, in one package, what is only available with analytics tools built on various popular document databases.

There is also an ad-hoc and prepared reporting tool for Elasticsearch called Kibana. It is already pretty impressive and is still being improved significantly. Unfortunately, it isn't all that easy to embed (for deployment) in a Java application but it is pretty easy to run on its own. For details see:

https://www.elastic.co/products/kibana

The Data Document and Data Feed features in Moqui Framework were designed for deriving data from the relational database to generate JSON documents. Initially the intent was to use them for searching and triggering things like sending email, but the general concept applies very well as an easy way to configure what will be pushed from the operational database to an analytic data store.

In short this combination of tools, already available in Moqui Framework (currently unreleased but available in the GitHub repositories), provide an excellent foundation for flexible, high scale reporting and analytics. It is easy to transform and feed data from the operational database to the analytic data store, and a wide variety of operations can be performed with high efficiency on this data.

How cool is this? With the combination of Moqui Framework and Elasticsearch you can put together complex reports that operate on huge volumes of data with very little effort.

Tuesday, September 23, 2014

Making Apps with Moqui - Book Now Available

My latest book is now available on Amazon.com:

http://www.amazon.com/Making-Apps-Moqui-Enterprise-Applications/dp/0692267050/

Making Apps with Moqui is the official documentation for Moqui Framework and includes a comprehensive summary of Mantle Business Artifacts. Starting with basic concepts and a tutorial to try things rights away, it builds to complete examples of end-to-end business processes including procure to pay, order to cash, and work plan to cash.


The framework topics cover data and service tier tools, user and system interfaces, security, and performance. With dozens of diagrams and screen shots, and thousands of lines of code and configuration examples, this book gives you ideas of what you can do with Moqui Framework and shows you how too. This includes things as simple as defining your data model with entities to more advanced things like building hierarchical data documents based on entity data and feeding them to other systems or indexing and searching the documents through simple configuration.


Learn how to easily build remote and local services that handle validation, security, transaction management, and much more. Build screens quickly with a wide variety of dynamic widgets and forms styled any way you wish, or even define your own widgets to use consistently across your applications. Handle large scale and milt-tenant systems. Track your application use and performance. Implicitly handle multiple languages, currencies and other localization details. Control access to resources across all tiers through flexible authc and authz configuration. 


Written by the founder of Moqui and Mantle, and an enterprise application architect with 15 years of open source and commercial experience, this book provides the most accurate and useful information available for building modern enterprise applications with some of the best open source tools and technologies.

NOTE: this book is also available for free download from http://www.moqui.org.

Thursday, June 19, 2014

Why I Work Without Pay on Free Software: Flow and Autotelic Creative Work

An Unexpected Experience of Flow


It was a hot day and I was sweaty from walking off the tension of a meeting that hadn't gone well. My thoughts at the beginning of the walk were scattered and disorganized, running through random details of the troubled project I had recently stepped up to manage. The effort to evaluate the project was bringing in more bad news every day. Complex issues and insufficient support for off-shore developers caused compounding delays in delivery. The project was expensive and critical to the future success and value of the organization. Pressure was high. By the end of the walk I remembered that none of these were new or unique problems, and I knew a few possible ways to address each one. I was still new in the position so while pressure overall was high I still had some time to get the project on a better course.

Feeling more calm I sat down to review the messages that had come in through the night and check on the status of the few dozen highest priority tasks. My mind jumped all over as I scanned through messages and tasks to get an idea of what was stuck and needed to be clarified and delegated. I felt stressed and scattered as I worked to wrap my head around it all. For so many of the blocked tasks I felt frustrated with no obvious answer of who to delegate them to. With so many of the issues no obvious solution came to mind.

Eventually I had scanned through all of them and sat back to consider the most important priorities of the project. I started to feel more relaxed and saw the big picture made up of all these little parts. I suddenly knew which stuck task to address first, just who to delegate it to, and what to write about how to approach the task. That done I jumped to the next task that a solution came to mind for, then to a message where a good answer to the question suddenly became clear.

Some time later I looked at the clock on my laptop and was surprised to see that almost two hours had passed. I didn't remember hearing any of the noise in the busy office, even though I was in an open cubicle. I had been interrupted a few times by people stopping by, but even those conversations seemed to fit in fine with what I was working on, and a couple even allowed me to address email messages in person instead of by written reply. I felt content and satisfied, even happy. I was no longer focused and the answers were no longer flowing, but it was okay. My mind felt somewhat empty. I sat for a moment and spaced out, and it felt nice.

As I got up to see who was interested in heading out for lunch I realized I had just experienced a deep flow. I was surprised because the conditions for it were so different from the quiet, focused, unpaid free software development work where I usually experience flow. This wasn't development work breaking down details of a single well defined goal, and not even a type of work that I particularly enjoy. It wasn't something I chose to do for its own sake, just something that had to be done. Still, the goal of the effort was clear as was the general purpose that the goal fit into. It wasn't a purpose I cared about deeply for its own sake, it was a paid effort, but the purpose was clear and I was part of making it happen. There wasn't a single task to focus on, and I didn't get to choose the tasks, but I did have flexibility over the order in which to handle the tasks. I also had control over how to handle each one, and at this point had a good enough knowledge of the project and the team so that along with general experience the challenge at hand was fairly level with my ability.

Not only did I get through far more in the morning than I had anticipated, and had felt stress about not staying on top of, but my tense and frustrating morning had turned into a pleasant and fulfilling experience. I was full of energy, ready to enjoy a lunch with colleagues, and well prepared to get through the many hours of planned meetings and moving along important objectives that remained in the day. I marveled at the power of flow. I mused over the wide variety of work that could be autotelic, that could be enjoyable and rewarding on its own enough to be intrinsically motivating and not just as part of a bigger external purpose or for extrinsic motivations.

My Free Software Based Career


In 2001 I started The Open For Business Project, now Apache OFBiz, a suite of ERP, CRM, and eCommerce applications (http://ofbiz.apache.org). Between then and 2010 I worked around 8,000 hours on it without pay. I then started a series of new free software projects that are a redesign and fresh implementation of a scope similar to OFBiz: Moqui Framework, Mantle Business Artifacts, and various other projects based on them (http://www.moqui.org). I have worked around 2,500 hours without pay on these projects, and anticipate thousands more.

Why would any sane person do such a thing? Setting aside the question of sanity, what about the whole issue of earning income for personal needs, family obligations and funding ventures? On top of all that I am a normal person with a wide variety of interests that require time and money such as off-grid living, exploring the wilderness on foot and motorcycle, constant reading and learning, meditation and mindful living, health and fitness, and even a little romance here and there.

When I started OFBiz my main motivation was to create something that filled a need I had seen in the two failed startups I had just worked for, and do so in a way that I could build a long-term career on it. Building commercial software does not qualify because if you part ways or the company goes under you can no longer effectively leverage the software you worked on in future contracts or business ventures. It is great to be paid to build something, but even better to build something that you can continue to use and improve no matter where you are working, or if you are working (for money).

These free software projects have been a great foundation for a career in enterprise software consulting. I have worked for some of the largest companies in the world, for fascinating and innovative startups, and everything in between. I have experienced business and culture many corners of North and Central America, Europe, and Asia. I have enjoyed a high level of independence and a good amount of work coming in through the free software projects, so I never really had to market myself in other ways. I get to work on the design and development of software that I am good at and enjoy, and not so much on the marketing and sales that I am neither good at nor enjoy very much, and yet still work as an independent consultant or in various incarnations of consulting organizations that I have tried over the years.

While these extrinsic motivators are great they are more of a side-effect and what it all comes down to is drive, to intrinsic motivation. This is not just something that drives me to work the long hours necessary for large and complex software projects, but is something that makes the work rewarding for its own sake. It is a powerful thing to experience the flow and satisfaction that comes from the combination of autonomy (of time, task, and team), challenge that often goes just beyond my existing skill and knowledge, and fulfillment in pursuing a greater purpose. These three are key to experiencing flow in creative efforts.

Doing creative work in a state of flow is both rewarding (pleasurable and satisfying) and nearly always results in effective, efficient work with excellent results. Working on free software projects benefits from a natural sense of purpose, frequent satisfying challenges, and a high level of autonomy. The experience of flow on free software or other more autonomous creative work carries over into day to day necessary work as well. People learn to get into flow and leverage it for improved results and satisfaction. This is one of the reasons why organizations allowing 20% time to work on whatever employees want see greater innovation and productivity overall.

So why do I work on free software? It is a foundation for a career, but more importantly it is something that I can tap frequently for the experience of flow. On the income side of things, it is easier to get into flow doing paid work based on the free software projects that I work on and in the ever growing set of business contexts that I am familiar with. This carries over into paid work and makes it easier for me to get into flow on those efforts. I pursue flow in other experiences like riding a motorcycle in the wilderness, adjusting speed and choosing terrain to challenge and engage myself just beyond my comfort zone (which varies day to day, and even during a ride), but the experience of flow in the context of a larger purpose and creative effort is far more satisfying and rewarding.

Books and Background


The best book for an introduction to the topic of intrinsic motivation and flow is:

Drive: The Surprising Truth About What Motivates Us by Daniel H. Pink
http://www.amazon.com/Drive-Surprising-Truth-About-Motivates/dp/1594484805

On an interesting side note this book features various stories about Atlassian, the company behind Jira, Confluence, and other software commonly used in projects based on OFBiz, and many software development projects in general these days. Atlassian has a policy of 20% time to work on anything desired, and various other policies that help foster and enable flow. There is some historical overlap between OFBiz and Jira, and Jira uses a customized version the Entity Engine from OFBiz.

For those interested in extreme sports and human performance in general, this is a great book:

The Rise of Superman: Decoding the Science of Ultimate Human Performance by Steven Kotler
http://www.amazon.com/The-Rise-Superman-Decoding-Performance/dp/1477800832

See also the Flow Genome Project (http://www.flowgenomeproject.co) which is based on this work, and in particular a great slideshow they put together that describes the concept of flow and trigger for it:

http://www.slideshare.net/StevenKotler/17-flow-triggers

Much of this is based on the extensive work of Mihaly Csikszentmihalyi. His most business-oriented book is:

Good Business: Leadership, Flow, and the Making of Meaning by Mihaly Csikszentmihalyi
http://www.amazon.com/Good-Business-Leadership-Making-Meaning-ebook/dp/B002J05GPS

For those who want to go deeper he has written various other books, and these are a couple of good ones to look into:

Flow: The Psychology of Optimal Experience by Mihaly Csikszentmihalyi
http://www.amazon.com/Flow-The-Psychology-Optimal-Experience/dp/0061339202

Creativity: The Psychology of Discovery and Invention by Mihaly Csikszentmihalyi
http://www.amazon.com/Creativity-The-Psychology-Discovery-Invention/dp/0062283251

The Experience of Flow


Flow is a powerful experience of full engagement and focus. It is a state of consciousness, attention, and awareness focused on a particular activity. The parts of the mind and body that are essential for the activity are fully engaged while the remaining parts of the mind and body are suppressed.

There are many aspects of the experience and the resources above have much information about not only what people experience, but also correlated changes in brain chemistry and electrical (wave) state and areas of the brain that activate and (more importantly) deactivate. One aspect is that time perception changes, usually a sense of time slowing down or simply no awareness of the passing of time. This correlates with transient hypofrontality, or more specifically change or suppression of the parts of the brain that monitor time and control your perception of it.

In some flow experiences even the parts of your mind that perceive your self will change and the boundary between what is you and what is not you will shift. In some experiences a tool may seem to become part of you so that no conscious effort is required to use the tool, it is like it is part of you. In other experiences you may feel more connected to the environment or to people around you, as if you are part of a greater whole (and this is an important aspect of team flow... yes, people can flow in groups too!).

The most critical aspect of flow is that self evaluation, your inner critic, turns off. You experience no value judgment and confidence becomes a non-question... the part of your brain that might doubt is simply turned off. Impulse control goes away along with activation of impulse focus on the task at hand. This feels like a steady stream of impulse and action based on impulse, a literal flow of continual impulse and action.

In a creative effort the parts of the brain that are then able to impulse and flow include the parts related to pattern recognition, linking ideas together, generating new ideas, risk taking, and detailed memory of the experience. When needed for the task there may also be increases in awareness, observation, and input acceptance and processing.

That is all the performance side of the experience, which will generally result in extrinsic rewards, but what about the intrinsic rewards? This is where the real motivation for the experience kicks in. During and following flow experiences the brain has an increase in norepinepherine, dopamine, serotonin, and endorphins. It is like being on low levels of speed, cocaine, MDMA, and opiates all at once... but totally natural. It is part of your mind's built in reward for your efforts in learning, growing, creating, and unleashing cultivated abilities.

To Flow and Beyond


There are steps before and after flow that you should be aware of to understand how flow fits into the overall creative experience (and other flow experiences as well). Below are descriptions of the 4 main phases. Chances are you will recognize these from past creative and other experiences, and understanding them you can start to recognize them in future experiences.

1. Struggle: When first beginning an effort you have a lot of mental noise and scattered thoughts jumping among the value,  purpose, and details of what you are doing. This is the phase to load relevant information into your mind by reading or conversing, to recall relevant information, and think about the goal you are trying to accomplish and the purpose of the effort. This is a necessary preparation for flow even if it does sometimes seem unpleasant. The inner critic that turns off during flow is active during the struggle phase. The dominant brain wave in this phase is the beta wave, and the most prominent chemicals in your mind are cortisol and norepinepherine. Focus your intentional effort on loading information and setting a clear goal and vision to reach that goal.

2. Clarity: In this phase the specific goal and vision for reaching the goal become more clear and you start to get your head around what you are trying to do. This is a key part of getting into flow and is the stage just before flow. If you don't have a clear enough vision of the effort you will never reach this phase and never transition into the flow phase. This phase may be brief or may last a while. To get here you need to relax and move your attention away from your thinking mind to let your mind do its thing. Intentional breathing, physical activity, or a simple distraction can help with this. You will eventually have a clear vision (not necessarily an image) of the goal, and what you need to do first becomes clear. Don't try to think through the whole effort in detail, just get the vision and the first step clear then start acting. Flow is all about engaging in that first step and letting the remaining steps "flow". What is literally happening is various parts of your mind are now doing the tasks you setup in phase 1 and coordinating with your conscious mind as needed by raising impulses for you to act on. Just focus on the first step of your effort, and let the rest come as you go. The prominent brain wave here is the alpha wave, and in this phase nitric oxide in the blood stream starts to increase, which in turn increases blood flow and the delivery of various chemicals to the brain that it needs to function optimally.

3. Flow: This is where the magic happens and you experience everything described in the previous section of this article. You start to feel fully conscious (associated with theta brain waves) and at the same time various parts of your mind become active while other parts of the mind that normally suppress these active parts become inactive (the inner critic turns off, time perception changes, etc). Most of the creative work moves to the background of your mind, your subconscious, leaving your conscious mind to act. You will start to efficiently process information and have more frequent and profound insights, which are associated with gamma brain waves. The more you can focus on the effort and let impulse and action flow the longer you will stay in this phase. This can be done with other people. Sometimes things others say and do will become a part of your flow, sometimes they will break your flow.

4. Detach and Recover: This phase always happens, and it is best if you allow the time and space for it instead of getting distracted or moving on immediately to something else. You need this phase to recover and learn from the effort, and to prepare for future flow. In this phase you may feel totally spaced out, empty minded. Your mind goes into this state as a natural way to recover from, and process what, happened during the flow phase. When your whole mind is focused on a certain task there may be memories, emotions, sensations, and observations that some part of your mind notices but that don't result in an impulse to your conscious mind because they are not relevant to the effort at hand. Those may come out during this phase. The prominent brain wave in this phase is the delta wave, and both serotonin and oxytocin may be elevated. This can be a very pleasant phase but can also be a let down from the high of flow. Either way it is a valuable phase for learning from your experience and preparing for the next time you do it. For those who meditate this is an excellent time to intentionally be mindful of what is going on in your mind and body. Your whole physiology is primed for mindful meditation in this phase, and doing so will maximize the benefits and the enjoyment that come from it.

Finding Flow


That sounds like a great experience, what can I do to get into flow, especially when doing something creative like developing software?

Ultimately flow is a mental state, something your mind does to respond to focused, high value effort. You can't force flow, but there is a lot you can do to create the conditions for it, to train yourself to more readily go into and stay in flow, and even change your brain chemistry and electrical state to facilitate it.

The Flow Genome Project has a list of 17 flow triggers, and there are three that are most critical to software development and similar creative efforts:

1. Sense of Control (autonomy + competence): Autonomy is all about independence and choice in the effort. The three main aspects of autonomy are time (when to do it, how long to spend doing it), task (what to do, how to do it), and team (who to do it with). Different people will consider different of these to be most important, and will be affected in terms of flow differently by each type of autonomy. Competence is the other side of the sense of control. With a higher level of experience and competence it is easier to get to a vision for how to go about the goal, and it is easier to quiet the inner critic that can block flow.

2. Strong Purpose and Clear Goal(s): A purpose that you care about will naturally engage you more. The more clear a goal is the more easily you will be able to put together a vision for reaching the goal and identify the first step to get there. The goal and your vision for achieving it is likely to change during an effort, and while it doesn't need to be perfect or even fully complete you need something to start with.

3. Challenge/Skills Ratio: For anything you might do you have a range of challenge (or intensity or stress) that you will handle well. There is a low end to this range below which you will have difficulty getting and remaining fully engaged. In other words you'll get bored and more easily distracted, it will be difficult to remain focused. There is a high end above which your skill and experience are not adequate and you will feel overwhelmed and doubtful or even fearful. The sweet spot, where it is most easy to get into flow, is a challenge level that is just a bit higher than your comfortable skill level. Flow happens best when you are fully engaged and this happens most naturally when you push yourself to new levels of performance and achievement.

Given these important conditions for getting into flow, how can you train and improve your ability to do so? One of the most powerful practices is meditation. The ability to be fully present in both time and place, focusing on the here and now, is cultivated with meditation and with practicing presence in everything you do. Even planning for the future or pondering the past can be done fully present, just chose the moments and focus on those things. If conditions are ideal for flow then it doesn't matter so much as your mind will shift into full presence, but it is much easier to learn to be present than to always try to setup perfect conditions or be limited to the perfect conditions. Being present reduces the impact of lack of control (both lack of autonomy and lack of competence), makes it easier to accept and embrace purposes and goals, quiets your internal critic, and widens the range of challenge that you will handle well.

When a challenge is well beneath your ability level increasing your attention will keep you engaged and present even though the intensity of the challenge doesn't push you there. When a challenge is beyond your comfortable ability being present will keep your mind from running to all the possible horrible outcomes, or the failures of the past. This is the essence of equanimity which is cultivated through meditation, various other mindfulness practices, and practicing presence and attention in everything you do. The ultimate experience of meditation is to detach your consciousness from your thinking mind, body, and emotions so that they are all in the background and your conscious mind is aware of them but not controlled by, or attached to, them. Creative flow is like a meditative experience in that you are more conscious and present in both, and being familiar with conscious awareness will help you get into flow.

One good book to get you started, if you are not already familiar with meditation, is:

Mindfulness Meditation in Everyday Life and Exercises & Meditations by Jon Kabat-Zinn
http://www.amazon.com/Mindfulness-Meditation-Everyday-Exercises-Meditations/dp/149151891X

Another tool that appears to have great potential, but that I have not personally tried, is neurofeedback training such as Brain Sport by SenseLabs (was Neurotopia).

Okay, that's great as an ideal, but what do I do to function effectively now while I'm practicing meditation and getting into flow? Be careful not to force flow, the extra stress will actually block it. Still, there is quite a bit you can do to "hack" flow. For example if a challenge is too easy you might intentionally make it more difficult, closer to the edge of your skill level, by doing it more elaborately, constraining it in some way (limit on time, size of result, etc), or even doing something passive at the same time such as listening to music or doing something physical (walking, stretching, etc). If a challenge is well beyond your skill level you can get help, train or study before you take it on, or identify the critical success factors and see if you can simplify the effort to something closer to the comfortable edge of your skill level.

A lack of autonomy or feeling of competence, or a purpose or goal that don't interest you, are a very different sort of challenge. These become exercises in acceptance. Acceptance is not about going along with things that are harmful to you or others, those are cases where you need to decide to be involved or not and what to do to set and enforce boundaries. Acceptance is about coming to terms with reality, letting reality be what it is without getting upset about it, and even without getting excited or happy about it. Just accept it for what it is, notice it in detail without any value judgment.

Once you understand a situation, internal or external, and are okay emotionally with what it is, then you can decide what to do with it and examine your feelings about it. By accepting a lack of autonomy and looking at the situation for what it is, you may find desire and fulfillment in the effort, and be able to flow fully with it even though it isn't ideal. By accepting a purpose or goal that doesn't interest you or even that seems undesirable at first, you may be able to see it and understand all aspects of it more clearly and start to feel interested and excited about it. It is natural when accepting something to be able to notice it more clearly, and natural when noticing something clearly to appreciate it for what it is and even feel gratitude and desire about it.

What if I want to get a boost while I'm doing inner work to be able to more fully and frequently flow? One way to suppress some of the same parts of your mind that are suppressed during flow is to use a tDCS device like the "foc.us gamer". This is a useful tool for concentration and can help with focus and turning off the inner critic, which can make it easier to get through the struggle phase into the clarity phase, which is often the most difficult part of getting into flow.

Another physical tool to help get into flow is nitric oxide (NO) boosting supplements. Nitric oxide naturally increases in the clarity phase and helps transition to the flow phase by increasing blood flow and delivering more of certain chemicals in the brain. There is a lot of research about nitric oxide and its role in mental and physical performance. There are common parts of fitness supplements that act as NO boosters such as L-Arginine and L-Citrulline. There are also more natural things that increase NO, including green leafy edibles, beets, garlic, cayenne, and good old fashioned exercise.

In general anything you can do to get the goal and details about it into your head, produce a vision for the effort, and focus intensely on the effort will help you get into flow. If you know a way to quiet the chatter in your mind effectively and safely then try that to help you get into flow. If you know ways to tune the intensity of a challenge to your skill and full engagement level, give it a try. Improving your ability to flow and the conditions for flow may involve working with others. These may be family, friends, or coworkers. Consider it an opportunity to improve your relationship with them, and maybe even get them interested in flow, presence, acceptance, and other things that will benefit them personally and professionally.

Breaking Down Goals for Ideal Flow


Software development involves lots of breaking down bigger goals into specific instructions and algorithm steps that a machine can execute. In a way it is ideal for getting into flow. Once you understand the goal and a general strategy for getting a machine to do that using whatever tools you have available you can get into the procedural flow of laying down one step at a time. Over time your mind gets so used to processes and rules that once you have the general idea of what needs to be done it is almost automatic for your mind to start breaking it down into structural and procedural chunks. Your mind trains well to send you a steady stream of impulses for you to act on as you tap out code.

For many of us this mental effort becomes so automatic that we can do it in spite of high pressure and distractions. We can also recover quickly and get back into flow after running into dilemmas and having to step back and observe the problem as a whole for a bit. In fact, we even get good at hopping between different levels of detail to work with higher and lower level processes and data structures and such, and getting into flow at varying levels of abstraction. There is a reason that once we develop some sound skill that we often thoroughly enjoy creating software.

What about higher level things, like gathering business requirements and designing user interfaces and other software development tasks where we don't have a nice clearly defined goal to work toward? What about when the task is to clearly define the actual implementation goals like screen designs, data structures, or business rules?

This is where levels of abstraction come into play. Business requirements are in effect just a high level, low detail abstraction of the software to eventually create, and preferably written in business terms as opposed to technical terms. Business requirements still live in the context of even higher level business objectives and rules. If you are using effective means of structuring and documenting requirements (such as those in HEMP, http://www.dejc.com/HEMP) then it will become a natural level of abstraction that you work with just as effectively and efficiently as code. It is a different sort of effort from coding as it is (or generally should be) a collaborative effort with business representatives. The potential when gathering requirements is to frame and direct the conversation in such a way that the whole group focuses on particular business objectives and processes and the whole group gets into flow together laying out the details of the business process.

Flow can be experienced in a wide variety of efforts. Even writing this article involved a few distinct periods of flow! Any large project or effort can be broken down into smaller efforts, some of which are efforts to define and break down the larger efforts. Over time breaking down larger efforts to optimize the opportunity for flow with each effort will be a natural part of your work planning and one of your flow facilitating skills.

Getting Started


Now you know what flow is and what to look for as you experience different phases of efforts leading up to flow, engaging in flow, and the detaching and recovering from flow. The best way to get started is to simply be more aware, start looking for the phases, sensations, and feelings of flow. Chances are you have experienced flow hundreds of times in your life, and maybe a few times as profound, sustained experiences that you recall clearly. Take a moment to sit in those memories and recall what it felt like, how you got there, similar experiences where it didn't work and what was different that got you fully engaged, present, and focused. In addition to experiencing flow in your work life and creative pursuits you can also experience it in personal efforts like pushing your limits in fitness or physical expression. Paying close attention to what you are doing and the sensations you feel as you do so can be a profound and euphoric experience as you get into flow, and as you get out of it.

You have also most likely had experiences of flow with other people, in personal or professional efforts. Those are special and powerful experiences, and teams that do this on a regular basis do amazing things and get profound satisfaction and fulfillment from it along the way. This is one thing people really enjoy about performing music, dancing, and other activities that involve people synchronizing and coordinating with constant engagement and feedback. In a business context this happens a lot during process-driven requirements gathering where people are focused on the overall business objective and work together to layout the steps to achieve it. It also happens during technical problem solving.

You can flow in a wide variety of efforts and circumstances, and some lend themselves to flow more than others. Practicing and enjoying flow in efforts that lend themselves to it will make it easier to get into flow in efforts where the conditions are less than ideal. These are often important things to do, critical for business operations and even certain parts of creative efforts and collaboration, and you can do them better and enjoy them more in flow. The experience may be one day a week where you work on whatever you want, in whatever way you want, for your employer. It may be unpaid contribution to free software projects like I have chosen, but it doesn't have to be, and it doesn't even have to be unpaid. The key is to set up the effort to use the triggers and follow the process for getting into flow.

Remember that going into flow states for important efforts is a natural part of the function and structure of your mind and body. Important is whatever you decide it is, or more accurately, whatever you tell your mind is important. You get to decide. Flow is the ideal way to learn and grow, to create and express, and your mind and body will reward you well for it.

Wednesday, January 8, 2014

Moqui Ecosystem Application Boost and Mentorship

One of the big objectives right now is to build out the application ecosystem based on Moqui Framework and Mantle Business Artifacts.

To help move this along I'd like to start building more community around existing projects (HiveMind and PopCommerce), and support developers working on these and other projects. The general idea is you work on functionality and I will review your work, offer feedback and training along the way, and if applicable (for HiveMind, PopCommerce, Mantle, etc) work with you to get your changes committed to the GitHub repositories for these projects.

For HiveMind I've been keeping a list of ideas for additional functionality here, and for those interested this is one good place to start:

https://github.com/jonesde/HiveMind/blob/master/WikiSpace/DEMO.cwiki

For PopCommerce I have been working on some general ecommerce functionality and there is a lot of work to do based on what is now in place for payment processor integrations (Authorize.net, Cybersource, PayPal, BitPay, etc), shippers (UPS, FedEx, etc), and tax calculation (Vertex, Avalara, etc). My plan is for each of these integrations to be a separate add-on component, BTW, and not part of Mantle or PopCommerce (but they would implement the now existing service interfaces and used the configuration touch points in Mantle).

There are also general things like a more real-world friendly checkout process, cart/order based promotions (price rules are already in place), support for products with variants, better browsing and parameterized search help (using augmented ElasticSearch/Lucene search strings like in HiveMind task search), and so much more.

These are just some ideas and I'm open to others as well, for HiveMind and PopCommerce and for other projects such as the EZBiz concept which may be split into multiple applications, and should probably have a different name too.

If you are interested in participating please send me an email (dej@dejc.com) describing what you'd like to work on and the relevant experience and skills you have.

Friday, December 27, 2013

Moqui Mini-Conf Jan 2014 Las Vegas

I now have some space in Las Vegas sufficient to host around 10 people for a conference, so planning to do some mini conference and training meetings here more regularly. The place is about 3 miles west of the LAS airport near highway 215. It has a good sized conference table and a connected room with projector setup that is a sort of lounge.



The first of these will be in late January 2014. I am open to feedback on dates and thinking of the 23-25 (Thu-Sat). The first day I'll offer general framework training and diving into more details based on questions and feedback during the training. The training can go into Mantle and HiveMind based on interest, but will be focused on Moqui Framework. The other two days will be general discussions and dev sessions in an un-conference format, in other words we'll start the conference with a huddle about what the rest of the conference will look like.

Some ideas for conference sessions and dev playing include the Mantle data model and services, playing with Drools rules (now used in Mantle for pricing, order shipping charges, and working on the tax calc side of it now) and jBPM workflows. The HiveMind project management and wiki functionality, ideas for improving it, and discussion about real-world use is another good topic right now.



Other fun topics might include ElasticSearch and a follow-up on the results of some ideas that came out of the last conference (the DataDocument, DataFeed, and DataSearch tools, and the Notification framework). For the notification framework a good topic of discussion would be using that with WebSocket, which I haven't done yet due to trouble finding a good library for WebSocket that is independent of the servlet container.

Given the legacy of Moqui/Mantle/etc in the Apache OFBiz project discussions around the two projects (differences, similarities, ideas for using them together, ideas for changing each to be more like the other, etc, etc) is another good topic for discussion.



The last conference was free, and the thanks goes to Hotwax Media for hosting the event. This conference will have a small charge of $50 for basic supplies and snacks and such, and $200 for the training day on Thursday for those interested.



Please comment on the Moqui Ecosystem LinkedIn group on general timing and interest, and contact me directly (dej@dejc.com) if you would like to attend.

Wednesday, November 27, 2013

Initial Releases of Mantle Business Artifacts and HiveMind Project Manager

Mantle Business Artifacts 0.5.0 is now available for download through GitHub here:

https://github.com/jonesde/mantle/releases/tag/release-0.5.0

HiveMind PM (Project Manager) 1.0.0 is now available here:

https://github.com/jonesde/HiveMind/releases/tag/release-1.0.0

It is exciting to release these first versions of these two projects. They represent the first significant body of business functionality based on Moqui Framework and make Moqui and Mantle an excellent alternative to Apache OFBiz and other open source ERP, CRM, and eCommerce open source projects and commercial products.

Mantle Business Artifacts 0.5.0

This is the initial release of Mantle Business Artifacts (version 0.5.0). It is based on Moqui Framework 1.3.2. This version covers procure to pay, order to cash, and work plan to cash processes including:

Purchase and Sales Orders (for goods, services, materials, etc; POs for inventory and equipment/supplies/etc)
Project, Task, and Request management with time and expense recording, billable/payable rates by project/task/client/worker/etc
Incoming and Outgoing Invoices with a wide variety of item types and an XSL:FO template for print or email
Automatic invoice generation for purchase orders (AP), sales orders (AR), project client time and expenses (AR), project vendor/worker time and expenses (AP)
Payments, both manually recorded and automatic through payment processing interfaces; applying payments to invoices
Fulfillment of sales orders (including basic picking and packing) and receiving of purchase orders
Inventory management including issuance and receipt, and inventory reservation for sales orders
Automated GL account posting of incoming and outgoing invoices, outgoing and incoming payments, payment application, and inventory receipt and issuance
General GL functionality for time periods, validation of transactions to post, time period closing
Balance Sheet and Income Statement reports (and basic posted amounts and account balance by time period summaries)

The automated tests are a great example of functionality and a good reference for where data ends up. They include complete flows with service calls and data validation for purchase orders (procure to pay), sales orders (order to cash), and billable projects with client and vendor/worker invoice/payment (work plan to cash).

HiveMind PM 1.0.0

HiveMind PM version 1.0.0 is the first official release. It is based on Moqui Framework 1.3.2 and Mantle Business Artifacts 0.5.0. This initial version of HiveMind Project Manager includes:

Vendor, Internal Org, Client, Worker, and other User administration
Rate setup for client billing and vendor/worker payout by project, worker, etc.
Project management
Milestone management and planning
Task management (hierarchical, associated with project and optionally milestone)
Flexible task searching and all-in-one task summary
Request management, including creating tasks to address the request
Time entry by task
Expense entry by project
Invoices and payments to pay vendor/worker for time and expenses
Invoices and payments to bill clients and receive payment
Invoice PDF
GL posting for invoices and payments
Balance Sheet, Income Statement, and posted balance/summary reports
Wiki with attachments for requirements and designs, linked to tasks and requests
Consolidated searching across projects, tasks, requests, and wiki
Simple email notifications for task, wiki, etc updates

The HiveMind-1.0.0-src.zip and HiveMind-1.0.0.zip include Moqui Framework and Mantle Business Artifacts and are ready to go. The HiveMind-1.0.0.zip file is built and has a Derby database preloaded with demo data, so just run it (gradle run, ant run, or java -jar moqui-1.3.2.war) and in your browser go to:

http://localhost:8080/apps/hm/

Latest Moqui Framework Release

Recent Releases Summary

It has been a while since I post about the latest in Moqui Framework, so here is a summary of major things in the last few releases (1.2.0, 1.3.0, 1.3.1, 1.3.2):

Moqui Framework 1.2.0 is a minor new feature release and a major quality improvement release. This release has undergone significantly more testing than previous releases because of a wider variety of functionality that has now been built and tested using the framework, and because of a unit testing effort (framework unit tests built using Spock).

The new features include popup menus (using the new jQueryUI menu widget), dynamic-options in XML Form drop-downs with dependency on other fields, automatic optimization of queries on view-entities to old select member entities necessary, support for UUID generated primary keys, expanded JCR support and an Example Content screen, and a number of small improvements that generally make the framework easier to use and more reliable.

Moqui Framework 1.3.0 is a major new feature and bug fix release. The major new features include the EntityFacade Data Document, Data Feed, and Data Search (based on ElasticSearch) features, the User Notification Message feature, and various improvements to XML Screens and Forms. This release is backward compatible with Moqui Framework release 1.2.0 with the exception that the StatusValidChange entity is deprecated by the new StatusFlowTransition entity.

Moqui Framework 1.3.2 is a minor new feature and bug fix release. The main new feature is a write-through per-transaction cache that effectively simulates the database for the scope of the transaction and does all DB updates with a single connection just before commit. This transaction cache is enabled on service definitions using the cache and force-cache options for the service.@transaction attribute. With this in place various Mantle services now run 2-3 times as fast with results validated by extensive automated tests.

Another set of new features covers internal profiling for artifact execution (entities, services, screens, transitions, etc). There are various forms of output available including a full artifact call tree with times, a consolidated tree to see own and total for artifacts in context, and a hot spot list by own or total time. This is similar to Java-level profiling tools but focuses on business level artifacts and with a low overhead gathers data always but only generates reports when needed.

OrientDB is now included in the framework by default instead of only in an addon component as before. This is useful as a general graph and document database and as an alternative through the Entity Facade for high-write entities like ArtifactHit.

Moqui Progress Commentary

While there are quite a few new features and bug fixes in Moqui Framework over the last few months, most of the work in the Moqui Ecosystem has gone into Mantle Business Artifacts and HiveMind Project Manager which are built on Moqui and are the main large scale projects based on it so far. I'll post separately about Mantle and HiveMind as both were released with a versioned release for the first time a few days ago.

The Moqui user community is expanding and the LinkedIn forum is starting to see more traffic. I am also starting to get more frequent calls and emails about new projects using them, and contracting and employment opportunities for projects based on Moqui (and often Mantle, and sometimes HiveMind).

Are you interested in getting more involved? The main thing Moqui and Mantle need now are applications built with them. These could be open source or commercial. The application should be something that benefits you, the better it is for you the better it will be for Moqui and Mantle as projects and products grow around them. Moqui and Mantle benefit from increased exposure, but also from feedback, bug reports, feature requests, patches, and so much more than existing efforts based on the projects have significantly helped with already (for around 2 years now).

What is coming next for Moqui Framework? This is often on my mind and is mostly driven by what is needed in applications that I am working on (Mantle, HiveMind, and PopCommerce now and then), technologies or features that people ask about and things that are just really cool or have huge potential to improve the flexibility and capability of enterprise automation systems.

One of these I have been researching more recently is the huge potential for rule and workflow tools in ERP apps. My interest in these goes back to the very early years of OFBiz, though neither the rule or workflow engines really worked out well and ended up being useful. Today there are extensive and very capable open source rule and workflow packages, and one (RedHat JBoss Drools and jBPM) where rules and workflow work together and share a knowledge base and such.

A couple of weeks ago I attended to Building Business Capability conference in Las Vegas and was inspired by how much progress has been made and all the great examples and case studies of applying these tools. This conference covers rules and workflow, and also requirements analysis and such that is another of my favorite topics in the enterprise software world.

I don't know when these things will show up in Moqui, but it will only be when there are real-world uses in place for them. Some early features in Mantle that I'd like to base on such tools include price calculation rules and order promotions rules (replacing the simple condition/action tables concept in OFBiz), order and invoice processing and approvals, and a more elaborate integration with HiveMind for managing the manual activities in workflow along with project tasks and such. There is potential for a lot more, but these would be good proof-of-concept sorts of features.