Re-Reference Post: Community versus Code and Glass Cathedrals

Some feeds missed this post (or weren't setup yet), so this is a short post to point back to it. There are some ideas of interest to the open source world, and as a bonus I've made an attempt at coining a new term: Glass Cathedral.

http://osofbiz.blogspot.com/2008/01/glass-cathedrals-and-community-versus.html

Glass Cathedrals and Community versus Code

Collaboration

From the proverbial "Day One" of thinking about OFBiz it was clear to me that I couldn't do it alone, and in the financial environment of the day (early 2001) it was also clear that there was no way I could raise the money necessary to make the dream a reality. That was especially true for me because I had nothing in my background other than a couple of years of research and failed commercial attempts to set me apart and make me of interest to prospective investors and partners.

Enterprise systems in general are large, complex things and require hundreds of thousands of man-hours to implement. On top of that with such a wide variety of businesses and real-world requirements designing something that would be flexible enough to be a good starting point and that would not require rewriting significant parts of the software would require a body of knowledge and experience that would be almost impossible to pay for. To start from scratch and build such a system would require a lot of collaboration.

The traditional means of collaboration in the commercial software world is to pay for a license and hopefully influence the vendor to add or fix things. There is little influence in this. Some vendors will accept payment for development of specific features, but for the most part customizations and specific features are built outside of the core product and never shared. This is a very limiting form of collaboration.

I worked with open source software before getting into all of this enterprise systems stuff as part of a company called Lineo that was creating products for embedded systems based on Linux, and in college as well. While doing R&D for an ecommerce startup I also ran into a number of open source alternatives, but mostly in infrastructure software, there just wasn't anything real happening in enterprise automation at the time.

The collaboration problem and the open source solution seemed the perfect match. Collaboration through a non-profit, open source effort seemed like a significantly more efficient way to go about things that collaboration through commercial licensing and the often disposable results of consulting engagements. With no financial entanglements collaboration can focus on solving problems and creating solutions with a long life span. Speaking of life span, the likelihood of survival for a piece of software is much greater when no one company can fall and take the software with it.

And there was more. I saw quickly that in addition to making the development of such a system from scratch possible, it also made the results of the development effort far better than any individual or static team working in isolation ever could. I started to see the lack of funding and the necessity of working on contracts to survive as a great benefit to the effort rather than a distraction from it. In short if I had personally had all the time and money I needed to site in a room and build this system, or even just the framework part of the system, it would not be anything like the real world need driven framework and application set that it is today.

The collaboration was just amazing. So many people suggested things over time and even though the majority of the suggestion never yielded anything immediately fruitful, there were so many that were critical and that pushed us in the direction we wanted to go by paths we hadn't even considered. I'm comfortable admitting that there is no way I could have built OFBiz the way it is now on my own. Over time I have also learned to accept that in many cases things I built and thought were great were inferior to existing alternatives or ideas about better ways of doing things. In the first 2-3 years of OFBiz we threw out as many lines of code as we kept. There was continual housekeeping and refinement, and there still is today even though these days the patterns and practices are more established and tried and proven, so the need for this is less frequent (on a large scale at least...).

After running OFBiz this way for years there were various people and organizations involved and this means of collaboration was proving to be very successful and stable. We were certainly not the only ones to run a software project this way and in 2006 some people from the Apache Software Foundation approached us, recognizing the similarities in philosophy, and inviting us to join the ASF. The more I read about Apache the more I came to see things this way as well, and the more I could see that they had the same emphasis on how to structure an open source project that I had come to. The great thing was that they had a much better organization and much better legal infrastructure (including the Apache License 2.0).

One of the main tenets of the Apache Software Foundation is that the community is the most important thing, and if you get that right then good code and other things naturally follow. In other words, if people are brought together in a way that enables minimally encumbered collaboration they will create something good, and something far better than if that collaboration had not been present.

Code

This may seem obvious to many people these days. These sorts of concepts have come a long way in recent years, even outside of the once fairly limited open source development circles. So why am I bringing this up now, and writing about all of this nearly 7 years after the inception of OFBiz?

The answer is that the principles and efforts of community driven open source projects have as much impact today as ever, and are a key to distinguishing efforts in the so called "open source" world.

There are some in the real open source world that don't focus on community and collaboration, and suffer because of it. One big potential problem for open source projects is a forked code base, or more importantly: a forked community. Sometimes a community degenerates or there never really was a strong community at all and a fork is appropriate.

In many cases though people fail to recognize the importance and value of the community and collaboration within it. For whatever reason they decide that they can do better on their own (often along with select others). They decide that the code is more important than the community.

In the OFBiz world there are various examples this failure to collaborate resulting in projects largely based on OFBiz but that have chosen to go in different directions. Many of these are open source projects, but organized and licensed in a way that fails to enable the collaboration necessary for success.

Glass Cathedrals

That's the main point of this blog entry, but I have a little more on my mind related to this and commercial dual licensing and the GPL license, and of course the dozens of variations on the license.

In general I agree with the ideals behind the GPL and the philosophy of Richard Stallman and others he works with on this. The idea of making software free and keeping it free is great, especially for certain types of software like infrastructure software that is used en-masse but not frequently customized (ie the improvements are nearly always generic enough to be shared). The reason that I originally pushed for the MIT license, and later the Apache License 2.0, is that these are more commercial and proprietary derivative work friendly licenses. When creating software that is meant to be customized it wouldn't be wise to legally or technically encumber those efforts.

What I really wonder is what Richard Stallman thinks about the practice of dual-licensing using the GPL and a commercial license. For these sorts of so called "open source" projects the only reason they want to use the GPL is its terms are the most onerous and are most likely to lead people to pay for a commercial license. In my opinion this is a deceitful corruption of valuable open source principles.

The development on these products (not really open source projects...) is generally done by a small group of paid designers and developers and there is little community interaction. Many choose to collaborate only financially instead of bringing their unique requirements and ideas to the table to enhance the core project. There isn't a lot of incentive to try to contribute back or collaborate when it is made clear from the beginning that there is one player who wants to own everything and that you don't really have a chance of being that player or competing with them in their own game.

The code becomes more important than the community. It's a bit like putting the cart before the horse in that there is a failure to realize the cause and effect and an attempt to get the affect (the code) with the cause (the collaboration in the community).

This is a major case where the use of the GPL breaks down collaboration instead of enabling it. This does happen in purely open source projects as well though, with GPL and other licenses too. Eric Raymond wrote a lot about this with his parables of cathedrals and bazaars. With cathedrals collaboration is limited and that is the biggest problem, whereas the collaboration possible in the bazaar is the real power in the model.

I would argue that this happens even with publicly developed open source projects if there is no incentive for or focus on enablement of collaboration, ie no real community that one can become a part of and influence and improve by just being positively involved. Some purely open source projects are this way, but most open/commercial dual-licensed products are this way. Even the "open source" parts are developed in a sort of "glass cathedral". People can see what is going on, and if they really want to they can walk in and participate, but most choose to stand outside and watch and in fact pay to not get too involved.

So, in closing, here's to Glass Cathedrals and the opportunity they give to community driven open source projects to differentiate ourselves. It also brings to mind that old saying... something about stones and those who live in Glass Cathedrals....

Thoughts on OFBiz Derivative Works, the PMC, and Opentaps Past and Present

In the email Si sent about resigning from the PMC he referenced reasons described in emails between Si and Andrew. I got with Andrew to review these and just wanted to comment briefly on them and what seems to be the issue.

From my point of view this is just the most recent incarnation of a problem that has been around for at least a couple of years now, and I've been more concerned over time about it but had decided to give it time and see how things have progressed. Initially this started with the financials module that was developed as an investment to be licensed for income until the development cost was recovered, but Si at one point decided he would rather put what Undersun had worked on back into OFBiz, and take full ownership of the rest to dual license (commercial and GPL open source). This resulted in something that is core to the mission of OFBiz being part of an outside project, which at the time seemed a reasonable compromise.

At the time and at various times since then I have had conversations with Si to warn him that this is not a tenable long term business model and eventually something would be developed as part of OFBiz to satisfy these requirements and encouraged him to consider licensing these back to OFBiz to be included in the project should a large user come along that had a sufficient budget and desire to re-develop it anyway.

Over time the scope of the financials application expanded to include refactorings and rewrite of functionality in OFBiz. The crmsfa component was also added to implement a SalesForce or SugarCRM type of sales management application (this was a piece that was originally designed for a client and proposed to be added back to OFBiz, but Si underbid the implementation to retain the IP and add it to the dual-licensed modules. More recently additional modules and rewrites have been added to the Open Source Strategies library of OFBiz extensions.

The reason these have become a problem is because they are within the established scope for OFBiz and this represent a competition for resources to build and maintain them. Various people have complained to me personally about this, and a few times various people have complained more generally on the OFBiz mailing lists. Things that have prompted these complaints include things like: requests on the OFBiz mailing lists for people to work on and contribute to opentaps/OSS components instead of to OFBiz, statements that represent the scope of OFBiz being limited to not conflict with what is being developed by Open Source Strategies (ie that it is a low level set of tools and application elements and not meant to be used by end users), statements about the nature of the OFBiz community and management process, etc.

I spent a bit of time with Andrew reviewing the emails between Andrew and Si. The main thing Andrew was complaining about was this conflict of interest and that it did not seem appropriate for someone to be on the OFBiz management committee who was, in order to further their own business interests, was trying to limit the scope of the project and recruit resources from the project community. Based on this Andrew expressed to Si that PMC members really should be advocates of the project and not send mixed messages, so it might be best to resign from the PMC if we was going to continue and expand these operations.

It should be noted that this was not a request by the OFBiz PMC in any official way, or even by Andrew as I understand it, but rather communication between two PMC members about a concern over something that is having an impact on the project. That said, as should be clear by my short history behind this conflict I definitely agree with Andrew that it is a problem and I think it is good as it is starting to have a greater impact that something was done.

Back to my thoughts on this... Part of the motivation for choosing the MIT license originally, and moving to the Apache 2.0 license as part of the ASF move, was to facilitate use of the software and participation in the project by a large number and variety of individuals and organizations (Person and PartyGroup in OFBiz lingo ;) ).

This means that according to the license what Open Source Strategies is doing is totally fine. From a community building perspective I think it is also okay as it is better than nothing. In other words if this is the only way we can get help from the people at Open Source Strategies, then so be it, we'll still take it! This is why I personally have decided not to do anything about this. Of course, the more they separate from OFBiz and fork certain components the less this will be true, and at some point the competition and dilution of resources that might contribute to OFBiz is more than the benefit to OFBiz from the indirect use.

I don't know if we've reached that point with either Neogia or Opentaps, which are the two main projects based on OFBiz that have forked certain parts, and extended independent of OFBiz. The reason I say that is that the OFBiz community is still strong and making good progress and is in fact growing rapidly and getting a lot of end user attention that is translating into even more contributor attention. We are also seeing a trend that those experienced with OFBiz are doing more dedicated work based on OFBiz and that is resulting in more contributions and more time spent on the project.

In general less involved users like this are great for the project, but really in order for the project to progress and fill the intended scope we need contributor individuals and organizations that practice a process of operation that meets client requirements by starting with generic development that goes back into OFBiz, and then continues with configuration or customization to meet the non-generic aspects of the requirements.

As for what is best for businesses trying to make a profit while working with the open source project, there are many models and many of them are viable and sustainable, and which one is best for the company the market (the customers) is yet to be seen. Here are some of them (a small selection):

1. dual licensed components or whole applications (commercial and generally GPL on the open source side, though opentaps using HPL which is more restrictive AND which is not truly "open source" as it is NOT OSI approved)
1.a. general purpose software
1.b. industry (niche industry or business type) specific solutions

2. commercial derivative works
2.a. general purpose software
2.b. industry (niche industry or business type) specific solutions

3. services and custom solutions

Based on talking with a lot of people there seems to be a continuum of opinion about open source versus commercial development models. To be clear, for this to make sense the goal is to create and maintain software that individuals or organizations can use. We're not talking about how to make a company that can produce a profit, but rather create an organization and collaboration model for the creation and maintenance and use of software. Some are totally on the open source side and think that is the answer to every question. Some are on the commercial side and believe _that_ is the answer to every question. Think of it kind of like the Kinsey scale, and just as with the Kinsey scale most people fall somewhere around one side or the other but not right at either side, and not so much right in the middle either. Perhaps someday this scale will have an official name and even an official diagnosis test to determine where someone's opinions lie on the scale. This might be an ente
rtaining research project.

At Hotwax Media, Inc. we have chosen #3 for now and we are growing quickly and having great success with a growingly impressive set of sites and companies we are working with. We believe, and are proving, that this is the best model for our company and for OFBiz itself as well. It allows us to stay close to the project, collaborate well with others, efficiently meet client requirements, and contribute major sets of functionality back to OFBiz. We are also working on extending OFBiz to implement software to support our own operations and make them more efficient, and that means even more goes back to the project, and is cheaper for us to develop and maintain.

I should make it clear that another commercial model that I've had in mind since the beginning of OFBiz is to create commercial derivative works for niche industries based on the generic open source software. The idea here is that the open source model doesn't work so well because in many industries the companies are too small and need too much software to customize it for a single business or for people from the various companies to collaborate on creating the software.

In other words many prospective users of OFBiz just can't effectively use the generic software as it is OOTB because it is way more than the need and probably too much for them to maintain as well. In order for those companies to use OFBiz they will need something customized for their type of business, and unable to collaborate directly in creating and maintaining that what makes the most sense is to collaborate through commercial means. Given the basis on open source generic enterprise applications these solutions can compete well because for a reasonable development expense something that targets a very specific market can be created. The customer gets a system that effectively automates all information management they need while still remaining highly affordable. Of course, this model is mostly theoretical as only very little has been done with it so far...

Moving the Blog

Just a quick note for now. This blog, which I obviously haven't updated in a while (since around the time the ASF effort ramped up), used to be hosted on the Undersun Consulting content site but that site will be going down soon because of the merger of Hotwax Media and Undersun Consulting that started in Dec 2006 and was official 1 Jan 2007.

Even when this was on the undersunconsulting.com site I was thinking of moving it to a site like this to make it easier to link to other blogs and manage it in general. I'll be adding to this over time, largely a commentary on what I see happening in OFBiz and mostly my point of view on where I'd like to see things go, or how I understand they might work best.

OFBiz Community Involvement and Jira (repost from 2005-04-15)

OFBiz is getting big. The code and features are getting big. The community is getting big. Because of this the way we have been managing things is becoming a bottleneck. The proposed solution, as presented here, is to organize and facilitate community involvement. This will happen primarily through Jira, and the process is described here.

Pretty much everyone involved in OFBiz does things on their own time and decides what they want to work on, or does things on an employer's or client's time, and prioritizes the needs and requests of those "what pay for da time."

I certainly try to handle incoming requests and issues, but at times it is simply more than I as an individual can handle. I really appreciate the help of Jacopo, and sometimes various others on these things, but more is needed...

With a tool like Jira we really can have more community involvement, and it would help a lot! There are dozens of outstanding issues in Jira, and various other bugs, issues, priorities, etc that are not in Jira. For things that aren't in Jira, we should push submissions there more, and I plan to start entering the things that I would like to work on there.

=========== THE KEY:

To help with my decisions on priorities for these and other things, I would REALLY appreciate community feedback. This helps the project serve the needs of those who are using it better, and it allows me to step out of the position of being that bad guy that tells people who have invested effort into something: sorry, I won't review it and put it in because in my opinion it is a lower priority than other outstanding issues.

How to help?

1. vote for tasks/issues in Jira that you think are important!
2. review submissions and improvements in Jira and comment on them
3. if you have a pet-peeve or other issue, submit it as a Jira issue with good detail, and then vote for it

This will help those that do the final review and commit to feel more comfortable that there are no major issues with the changes and additions, and if there are issues to get them resolved before the final review and commit effort begins.

If you have an issue that you think is important and no one is looking at it, don't ask me or the other commiters to take care of it, send a message to the users or dev mailing list and ask people to review it and vote for it.

There have been various discussions about this over time, and I think the project is to the point where not doing things this way is preventing growth and progress...

I'm very interested in hearing feedback on this, and even MORE interested in seeing feedback and voting going on in Jira.

Thanks to everyone for all of your help. I know you have put a lot of effort into contribution to the project, and it is these efforts that makes OFBiz what it is.

Collaboration: The Key to Success (repost from 2005-04-15)

Sometimes it is difficult to know how to go about leveraging an open source application like OFBiz in your company. This is especially difficult when people involved are not used to working on projects with larger teams and overlapping objectives. There are some misunderstandings and very tempting ways of doing things that everyone should be careful of, and it really comes down to effective collaboration....

One misunderstanding I see frequently is concern over occasional bugs that are discovered or changes being made to things you are basing something on, or in general not trusting the code or not liking the fact that additional work is required in some cases to make your own stuff continue to work based on changes, or be delayed because of a bug in something you need. Based on my experiences digging into these concerns, they are phantom concerns and is not the real issue. The real issue is the thought that there is a need for some sort of a stable set of code to work from, or that a strict isolation from the open source project is desired.

BE CAREFUL, THIS IS DANGEROUS! I say that in all caps because I really mean it. The fact of the matter is that the company is engaging in a development project, or more a customization project. You have a high level of collaboration happening within any company or group doing an OFBiz-based project, with sometimes one or two or sometimes dozens of people who will be making and committing changes to the company source repository. The aspect of this that I am talking about that is dangerous is the idea the you want to take a "snapshot" or somehow theoretical perfect set of code to use as a basis for your efforts. Unfortunately this theoretical perfect starting point doesn't exist, not in OFBiz, and not in any other piece of software open source or commercial.

A better way to look at this is that each person is working in collaboration with a number of other people, all of whom are working on projects that in one way or another move the software in the desired direction. The key thing to realize when working with an open source project like OFBiz is that there are many more resources than just those inside the company that are helping with a larger effort. Some of that larger effort (ie the set of all efforts going into OFBiz) is of interest to the company, and some of it is not. From an enlightened point of view, a LOT of things that might at first seem unimportant to the company and its specific project really are important and will over time save the company a lot of money, and the people involved with the project a lot of effort.

In other words, the fact is that OFBiz is a very large piece of software with many features and it is only because of those features that this specific project for this specific company can be a success for the effort being put into it. Software like OFBiz requires a LOT of work for maintenance and even though the company is putting a lot of resources into the effort, it is almost universally not near enough to maintain the entire set of software, fix bugs, create new solutions to problems that come up, modernize old code to make customization more efficient, and so on. Various groups have taken the "snapshot" approach and eventually they lose out on the thousands of hours of effort that result in improvements in OFBiz every single month.

The downside to this is that other people will work on new features that go into the open source project but that the company doesn't care about (not at that moment anyway), and in some cases those can conflict with what the company is doing. In most cases evaluation of the conflict results in a significant improvement in what the company ends up doing. But still, yes, there is some flexibility and reaction and conflict resolution that is required in the company to handle the side effects of what others in the community are doing. Still this cost is usually not even in the same order of magnitude (and often 2 orders of magnitude, in other words 100 times) separated from the effort that would be required to reproduce in the company everything that is going into the open source project that is of value to the company.

Some strategies to consider... Unless there is a specific reason for it, like getting a specific bug fixed or getting a new feature in place, it is best to only "sync up" with the open source project once every week, or every other week even. This is a very common tactic used in large development teams that are segmented into different teams with different purposes. In order for each team to move fluidly, they don't necessarily stay in sync continuously, but instead work in their smaller groups and continuously stay in sync there, and then frequently (but not continuously) combine all efforts into a single code base. Once the "feature freeze" happens (before the testing phase in preparation for a deployment) this process changes and typically only bug fixes are introduced, and everything is continuous.

There is a good book that I found recently on this topic that explains how to facilitate these sorts of things using Subversion. It is called: "Pragmatic Version Control, Using Subversion" by Mike Mason. There is also a CVS version of this book, but note that because things in CVS are less flexible (like not being able to move files and directories, for example), the approaches for CVS are much more difficult.

In general the key to success for any large project is: collaboration. When working with open source software effective collaboration is even more important, and at the same time the open source way of creating and maintaining software is a major facilitator of collaboration.

The trick is for people to recognize some of these key concepts and then embrace collaboration, and not fight it or fear it.

Some "Rules" for Collaboration that might be good to present and discuss and frequently remind people of in an large project:

1. understand that the effort is much larger than any one person can accomplish
2. recognize the value of what other people are doing in the larger scope of the project
3. recognize that other efforts may at times require modification of the same files that you are modifying, and therefore sometimes introduce conflicts that will require your effort to resolve, and all for the better good and overall progress
4. accept that you and everyone else WILL make mistakes along the way, and give people a chance to correct their own mistakes by letting them know about the problem you found; note that sometimes the problem is caused by multiple people, and multiple people will have to work together to solve them

I hope that I am speaking to the real issue, and the real fears and anxieties that people are facing. From feedback I hear every day this is a VERY common fear and misunderstanding.

OFBiz User Interfaces General Thoughts (repost from 2005-04-06)

There are 2 main themes that seem important for the progression and general design of OFBiz:

1. Easier administration of customer/public facing content, preferably through the browser or some other real-time remote server interaction (like webdav)
2. Design goals of the administrative user interfaces, ie generic and "complete" versus task/goal oriented (a good point brought up in this last message)

Thanks to Ean Scheussler for bringing up these issues on the OFBiz dev mailing list.

Some thoughts/comments on these, including why some things are the way they currently are in OFBiz or rather the intentions during the creation of these things:

1. This has come up a few times in the history of OFBiz, and in my opinion being able to modify certain parts of public and customer facing sites it's a great idea and one that a lot of people ask for. At one point we were in discussions with eInnovation in Cincinnati to integrate WSPublisher (now OpenEdit) into OFBiz, but they eventually pulled back and were flaky over the licensing so it never worked out...

The Content Management stuff was brought up in this context, and some effort has been put into moving it in this direction, but it is most certainly not finished and the tools that were meant to facilitate this application of the infrastructure was never really cleaned up for real use. What I'm referring to is the Layout editor in the Content Manager, and certainly not the raw Content and DataResource pages (which have a couple of design issues, and a couple of things are broken, but they generally work...).

This approach is very easy for small companies, or sites that are administered by a small group of people. For larger groups and more constraints things get hairy... Permissions for editing and viewing, revision management, trial and production deployments, etc, etc.

This is great for managing some content on a site, but in my opinion quite a bit of a pain for certain other things, especially things that are very dynamic. In general some things make very good sense to put in the database, but I hate having code in the database for the most part because it is so difficult to keep synchronized with other things, to edit and do global searches, etc. Of course, all of this can be done but now you don't have any of the common tools to use, you have develop your own for every single little thing...

2. There's no question that in general the administrative applications in OFBiz are VERY generic and are not helpful for guiding the user through specific tasks. Some things will guide you through small sets of steps, but for the most part people do seem to have a hard time getting used to where to find things or where to get started with specific tasks they need to do.

The problem with the approach of trying to define simple and limited user interfaces that are oriented to specific tasks is that (like Ean started hinting at)

1. they become highly redundant (ie lots of screens that deal with a specific entity like ProductFeatureAppl or WorkEffortRole, or even worse for the "top level" entities like Product, Party, and Content)
2. you end up with TONS of them in a project that is trying to be fairly universally applicable like OFBiz which means lots of work to develop and maintain, more confusion when trying to figure out not only how but WHAT to customize...

My thoughts on this right now are that we should keep the UI's fairly generic in OFBiz and to facilitate their use by writing role and task oriented documentation that in essence describes where to go and what to do when you want to accomplish certain tasks or goals.

That is the intention behind the design and organization of the "Application Overview for Users" document in the Undersun on-line documentation. And yes, we do charge for that because it is NOT necessary to use OFBiz but it does make it easier for various types of users to use, and we have a technical writer that is more or less dedicated to this. Actually, so far there has been so little money and so many complaints about a company that is not affiliated with OFBiz doing such a thing that I'm tempted to drop charging for it and just open it up. I don't for 2 reasons: 1. spite over the complaints, 2. there would be so much load on the server that we couldn't live with what we have now and would need another server (or 2).

Another thing related to this, and going right back in some cases to the commercial side of the world... is the idea of creating derivative works of OFBiz that basically amount to applications that ARE designed to be role and task oriented with user interfaces very customized to a certain way of doing things and for managing certain types of data. Creating these things basically amounts to creating forms with a lot of "ignored" fields, and stringing them together in wizard type things, plus creating special ways to view or visualize the data, which usually have links that represent starting points for tasks related to specific data.

This is a nice approach that makes things much easier for users. The "specialized" folder in OFBiz has some applications going in this direction that are really pretty good. This also opens up a huge world of commercial opportunity to create full-featured applications for specific types of businesses, or really full-featured solutions that can include support and other things that most companies need but can't do for themselves, and often can't afford in face of the large licensing fees required to acquire the access to the functionality they need.

Okay, now I'm blabbing on....