Sunday, May 27, 2007
Even when this was on the undersunconsulting.com site I was thinking of moving it to a site like this to make it easier to link to other blogs and manage it in general. I'll be adding to this over time, largely a commentary on what I see happening in OFBiz and mostly my point of view on where I'd like to see things go, or how I understand they might work best.
OFBiz is getting big. The code and features are getting big. The community is getting big. Because of this the way we have been managing things is becoming a bottleneck. The proposed solution, as presented here, is to organize and facilitate community involvement. This will happen primarily through Jira, and the process is described here.
Pretty much everyone involved in OFBiz does things on their own time and decides what they want to work on, or does things on an employer's or client's time, and prioritizes the needs and requests of those "what pay for da time."
I certainly try to handle incoming requests and issues, but at times it is simply more than I as an individual can handle. I really appreciate the help of Jacopo, and sometimes various others on these things, but more is needed...
With a tool like Jira we really can have more community involvement, and it would help a lot! There are dozens of outstanding issues in Jira, and various other bugs, issues, priorities, etc that are not in Jira. For things that aren't in Jira, we should push submissions there more, and I plan to start entering the things that I would like to work on there.
=========== THE KEY:
To help with my decisions on priorities for these and other things, I would REALLY appreciate community feedback. This helps the project serve the needs of those who are using it better, and it allows me to step out of the position of being that bad guy that tells people who have invested effort into something: sorry, I won't review it and put it in because in my opinion it is a lower priority than other outstanding issues.
How to help?
- vote for tasks/issues in Jira that you think are important!
- review submissions and improvements in Jira and comment on them
- if you have a pet-peeve or other issue, submit it as a Jira issue with good detail, and then vote for it
This will help those that do the final review and commit to feel more comfortable that there are no major issues with the changes and additions, and if there are issues to get them resolved before the final review and commit effort begins.
If you have an issue that you think is important and no one is looking at it, don't ask me or the other commiters to take care of it, send a message to the users or dev mailing list and ask people to review it and vote for it.
There have been various discussions about this over time, and I think the project is to the point where not doing things this way is preventing growth and progress...
I'm very interested in hearing feedback on this, and even MORE interested in seeing feedback and voting going on in Jira.Thanks to everyone for all of your help. I know you have put a lot of effort into contribution to the project, and it is these efforts that makes OFBiz what it is.
Sometimes it is difficult to know how to go about leveraging an open source application like OFBiz in your company. This is especially difficult when people involved are not used to working on projects with larger teams and overlapping objectives. There are some misunderstandings and very tempting ways of doing things that everyone should be careful of, and it really comes down to effective collaboration....
One misunderstanding I see frequently is concern over occasional bugs that are discovered or changes being made to things you are basing something on, or in general not trusting the code or not liking the fact that additional work is required in some cases to make your own stuff continue to work based on changes, or be delayed because of a bug in something you need. Based on my experiences digging into these concerns, they are phantom concerns and is not the real issue. The real issue is the thought that there is a need for some sort of a stable set of code to work from, or that a strict isolation from the open source project is desired.
BE CAREFUL, THIS IS DANGEROUS! I say that in all caps because I really mean it. The fact of the matter is that the company is engaging in a development project, or more a customization project. You have a high level of collaboration happening within any company or group doing an OFBiz-based project, with sometimes one or two or sometimes dozens of people who will be making and committing changes to the company source repository. The aspect of this that I am talking about that is dangerous is the idea the you want to take a "snapshot" or somehow theoretical perfect set of code to use as a basis for your efforts. Unfortunately this theoretical perfect starting point doesn't exist, not in OFBiz, and not in any other piece of software open source or commercial.
A better way to look at this is that each person is working in collaboration with a number of other people, all of whom are working on projects that in one way or another move the software in the desired direction. The key thing to realize when working with an open source project like OFBiz is that there are many more resources than just those inside the company that are helping with a larger effort. Some of that larger effort (ie the set of all efforts going into OFBiz) is of interest to the company, and some of it is not. From an enlightened point of view, a LOT of things that might at first seem unimportant to the company and its specific project really are important and will over time save the company a lot of money, and the people involved with the project a lot of effort.
In other words, the fact is that OFBiz is a very large piece of software with many features and it is only because of those features that this specific project for this specific company can be a success for the effort being put into it. Software like OFBiz requires a LOT of work for maintenance and even though the company is putting a lot of resources into the effort, it is almost universally not near enough to maintain the entire set of software, fix bugs, create new solutions to problems that come up, modernize old code to make customization more efficient, and so on. Various groups have taken the "snapshot" approach and eventually they lose out on the thousands of hours of effort that result in improvements in OFBiz every single month.
The downside to this is that other people will work on new features that go into the open source project but that the company doesn't care about (not at that moment anyway), and in some cases those can conflict with what the company is doing. In most cases evaluation of the conflict results in a significant improvement in what the company ends up doing. But still, yes, there is some flexibility and reaction and conflict resolution that is required in the company to handle the side effects of what others in the community are doing. Still this cost is usually not even in the same order of magnitude (and often 2 orders of magnitude, in other words 100 times) separated from the effort that would be required to reproduce in the company everything that is going into the open source project that is of value to the company.
Some strategies to consider... Unless there is a specific reason for it, like getting a specific bug fixed or getting a new feature in place, it is best to only "sync up" with the open source project once every week, or every other week even. This is a very common tactic used in large development teams that are segmented into different teams with different purposes. In order for each team to move fluidly, they don't necessarily stay in sync continuously, but instead work in their smaller groups and continuously stay in sync there, and then frequently (but not continuously) combine all efforts into a single code base. Once the "feature freeze" happens (before the testing phase in preparation for a deployment) this process changes and typically only bug fixes are introduced, and everything is continuous.
There is a good book that I found recently on this topic that explains how to facilitate these sorts of things using Subversion. It is called: "Pragmatic Version Control, Using Subversion" by Mike Mason. There is also a CVS version of this book, but note that because things in CVS are less flexible (like not being able to move files and directories, for example), the approaches for CVS are much more difficult.
In general the key to success for any large project is: collaboration. When working with open source software effective collaboration is even more important, and at the same time the open source way of creating and maintaining software is a major facilitator of collaboration.
The trick is for people to recognize some of these key concepts and then embrace collaboration, and not fight it or fear it.
Some "Rules" for Collaboration that might be good to present and discuss and frequently remind people of in an large project:
- understand that the effort is much larger than any one person can accomplish
- recognize the value of what other people are doing in the larger scope of the project
- recognize that other efforts may at times require modification of the same files that you are modifying, and therefore sometimes introduce conflicts that will require your effort to resolve, and all for the better good and overall progress
- accept that you and everyone else WILL make mistakes along the way, and give people a chance to correct their own mistakes by letting them know about the problem you found; note that sometimes the problem is caused by multiple people, and multiple people will have to work together to solve them
I hope that I am speaking to the real issue, and the real fears and anxieties that people are facing. From feedback I hear every day this is a VERY common fear and misunderstanding.
There are 2 main themes that seem important for the progression and general design of OFBiz:
1. Easier administration of customer/public facing content, preferably through the browser or some other real-time remote server interaction (like webdav)
2. Design goals of the administrative user interfaces, ie generic and "complete" versus task/goal oriented (a good point brought up in this last message)
Thanks to Ean Scheussler for bringing up these issues on the OFBiz dev mailing list.
Some thoughts/comments on these, including why some things are the way they currently are in OFBiz or rather the intentions during the creation of these things:
1. This has come up a few times in the history of OFBiz, and in my opinion being able to modify certain parts of public and customer facing sites it's a great idea and one that a lot of people ask for. At one point we were in discussions with eInnovation in Cincinnati to integrate WSPublisher (now OpenEdit) into OFBiz, but they eventually pulled back and were flaky over the licensing so it never worked out...
The Content Management stuff was brought up in this context, and some effort has been put into moving it in this direction, but it is most certainly not finished and the tools that were meant to facilitate this application of the infrastructure was never really cleaned up for real use. What I'm referring to is the Layout editor in the Content Manager, and certainly not the raw Content and DataResource pages (which have a couple of design issues, and a couple of things are broken, but they generally work...).
This approach is very easy for small companies, or sites that are administered by a small group of people. For larger groups and more constraints things get hairy... Permissions for editing and viewing, revision management, trial and production deployments, etc, etc.
This is great for managing some content on a site, but in my opinion quite a bit of a pain for certain other things, especially things that are very dynamic. In general some things make very good sense to put in the database, but I hate having code in the database for the most part because it is so difficult to keep synchronized with other things, to edit and do global searches, etc. Of course, all of this can be done but now you don't have any of the common tools to use, you have develop your own for every single little thing...
2. There's no question that in general the administrative applications in OFBiz are VERY generic and are not helpful for guiding the user through specific tasks. Some things will guide you through small sets of steps, but for the most part people do seem to have a hard time getting used to where to find things or where to get started with specific tasks they need to do.
The problem with the approach of trying to define simple and limited user interfaces that are oriented to specific tasks is that (like Ean started hinting at)
1. they become highly redundant (ie lots of screens that deal with a specific entity like ProductFeatureAppl or WorkEffortRole, or even worse for the "top level" entities like Product, Party, and Content)
2. you end up with TONS of them in a project that is trying to be fairly universally applicable like OFBiz which means lots of work to develop and maintain, more confusion when trying to figure out not only how but WHAT to customize...
My thoughts on this right now are that we should keep the UI's fairly generic in OFBiz and to facilitate their use by writing role and task oriented documentation that in essence describes where to go and what to do when you want to accomplish certain tasks or goals.
That is the intention behind the design and organization of the "Application Overview for Users" document in the Undersun on-line documentation. And yes, we do charge for that because it is NOT necessary to use OFBiz but it does make it easier for various types of users to use, and we have a technical writer that is more or less dedicated to this. Actually, so far there has been so little money and so many complaints about a company that is not affiliated with OFBiz doing such a thing that I'm tempted to drop charging for it and just open it up. I don't for 2 reasons: 1. spite over the complaints, 2. there would be so much load on the server that we couldn't live with what we have now and would need another server (or 2).
Another thing related to this, and going right back in some cases to the commercial side of the world... is the idea of creating derivative works of OFBiz that basically amount to applications that ARE designed to be role and task oriented with user interfaces very customized to a certain way of doing things and for managing certain types of data. Creating these things basically amounts to creating forms with a lot of "ignored" fields, and stringing them together in wizard type things, plus creating special ways to view or visualize the data, which usually have links that represent starting points for tasks related to specific data.
This is a nice approach that makes things much easier for users. The "specialized" folder in OFBiz has some applications going in this direction that are really pretty good. This also opens up a huge world of commercial opportunity to create full-featured applications for specific types of businesses, or really full-featured solutions that can include support and other things that most companies need but can't do for themselves, and often can't afford in face of the large licensing fees required to acquire the access to the functionality they need.
Okay, now I'm blabbing on....
One thing that I find interesting though is that about 30 OFBiz based ecommerce sites went live (that we are aware of) without any interaction from Andy or me, and without the conveniences that are now available such as the Basic Production Setup document, the training videos, the Undersun online documentation, etc.
I imagine that some of the people working on these learned things from the mailing lists, but my guess is they took the approach of considering the code base to be a rich resource of examples, and used those examples to figure out how to do things... That is mostly a guess though...
In fact, my opinion on this topic is that there is a LOT of functionality in OFBiz, and while you can learn the basics of the framework and tools and a week (not easy, but possible; and only a general understanding for sure, not all details available), it can take many months to learn about all of the data model and the logic and the user interfaces, basically all of the business level stuff. So, no matter what materials exist it will take time for your brain to soak everything up, and the examples are by far the richest and most dense resource for doing so.
All of the other documentation and videos and such are really only meant to make it easier for you to get started. They won't finish the process, and no materials ever will, because your needs may be pretty unique and it is impossible to predict what they will be and write a document that gives step by step instructions on how to do everything that anyone might want to do. At some point, some live human effort is necessary.
So, the documentation offerings from Undersun are NOT necessary to get going with OFBiz. Dozens of people are doing great things without ever looking at those. However, they can help you to get started and that is their intent.
There are lots of books that could be written about OFBiz to address different needs and audiences. And these really are very different audiences... We have talked with a potential publisher about doing a book about getting an ecommerce or general retail business going with OFBiz, though from this discussion it sounds like what is desired is something about development using the OFBiz framework.
One of the problems is that OFBiz is not corporate backed or sponsored by any company that press and publishing people like to toss out to make them look better, and when we talk about volume in terms of the current community the conversation seems to move towards self publishing or doing a PDF or other online book, or considering it as an investment and hoping that if done with a concrete enough real world benefit people might pick it up just for that, in spite of there being so little demand for it in advance.
I guess the main issue is that even though OFBiz can be used effectively in pretty small businesses, and as it is more complete that is improving weekly, but a fair amount of knowledge and skill is still needed to really do something with the software. So, we might be able to sell 100 or if we were lucky 200 copies of a book over the first few months to a year if it were available now. The only way to increase that would be for a publisher to decide they like OFBiz and spend some money on marketing the book, part of which would be the printing and distribution costs and trying to get buyers from major book retailers to buy it.
So, you're probably thinking: come on! It's not that hard! Look at all of these little open source projects that are doing it! That may be true, but consider how many technical journal articles and such those "little" projects have published before they can even consider doing a book... Even though users and potential users of something like OFBiz love the idea of open source enterprise software, it still isn't very popular in the incumbent technical and enterprise software business worlds. In fact, there is a little bit of hostility to the whole notion and it's hard to be taken seriously since so many people doubt that what we are doing can be done.... As the project progresses and more live sites and case studies are available as proof this is changing, but it is being totally driven from the edge, or in other words the end-user companies. There isn't much interest yet from the current big software providers.
In that sense a more limited scope architectural tool that doesn't require potential users to give up so much of what they love and are personally invested in is a LOT easier to sell to the world, and to the likes of Sun and Apache and various other groups that are heavily into the theory of enterprise software and building out of the infrastructure. For most developers they don't want a complete solution, because even if they did they couldn't sell it internally or to the management of their company, unless they are in very particular and somewhat desperate circumstances.
So, the fact of the matter is that OFBiz is being driven from the business side of things. This is one reason we'd like to create a somewhat separate framework project that can grow on it's own merits without the implication that using the entire data model and applications is necessary...
In the mean time, OFBiz is growing and progressing fairly nicely being driven by business needs, and this is being served well by face to face training, assistance with custom development, and now for the lower end training videos and reference documents and end-user oriented documentation.
So, no, it isn't nearly as simple as a small scoped project like Spring or Struts or Hibernate. Though in a way the framework is simpler that those combined once you get a full multi-tiered web-capable stack in place... You can't compare learning the OFBiz framework to learning Spring or Struts or Hibernate or the Servlet API, it is really more comparable to using all of them together, or whatever tools of the day that you like the idea of using in your master plan.
That's the problem with OFBiz. We HAVE to have a master plan for the architecture because we have hundreds of thousands of lines of business artifacts that depend on it, and it has to be efficient. So in order to use OFBiz you kind of have to be willing to go with the tools in use, either that or throw out the existing business and application artifacts and start over on your own, perhaps with just the data model or parts of it or something...
So, will there ever be a book published about OFBiz from a major publisher that sells thousands or tens of thousands of copies? Yes, I really think there will be, but the project and the current community just aren't ready for it yet.
I want to make sure it is clear that these are my own thoughts and opinions and observations. While based on fact and experience, there is naturally a lot of speculation in here...
One of the big principles that feeds my desire to see open source software succeed is that centralization of power is a dangerous thing. With it one man, often influenced by others, but still one despot can cause pain and suffering in the lives of thousands or millions of people. History both ancient and modern is full of examples of this, on small scales and on large. In this age there are despots both political and economic. Consider that large companies these days control sufficient wealth and in some cases power to have an effect for good or bad on hundreds of thousands or even many many millions of people.
Earlier today Al Byers sent me an article in the Salt Lake Tribune that I found interesting and thought provoking because of personal experiences I have had with this. Here is the link:
I live in Utah, or more specifically in Utah Valley, where much of this is taking place. I know a lot of people who work for or have worked for Novell, WordPerfect, and various smaller technology companies here, some of the most notable these days being Canopy Group companies.
For a long time in Utah Valley the economic basis was fueled by software companies such as WordPerfect and Novell. There are other major economic influences here, like over 20% of the population are students, many of whom are from out of town. There have also been other forces here such as a large steel plant and such, but the software industry has been a big deal here in the past and it's decline is having a significant effect here.
Having met Ray Noorda a few times I was really surprised by the actions and direction of the Canopy group, especially by the use of their resources to try to go after more honest and hardworking corporations and individuals with legal attacks, so this explains a lot. It reminds me of a few meetings various years ago when I was working at Lineo that both Ralph Yarro and Ray Noorda attended. Ray was always very quiet and didn't seem to overly involved, almost there as an observer. Ralph has a pretty forward and overbearing personality.
Their dress is also very different. Ray's idea of dressing up seems to wearing old church clothes and black sneakers to at least be the right color. I've seen him at church and in business meetings this way. He lives in a fairly modest house, about like the one my parents live in, and doesn't drive fancy cars or anything. He regularly attends church (the same congregation where my mother-in-law and others in my wife's family attend). His family has only seen enough of them wealth to make sure they are comfortable, but he does not go after the comforts and luxuries of the world nor has he spoiled his children with such.
Ralph Yarro seems to be somewhat the opposite. I don't know him as well and haven't seen him as much but he seems to like to dress expensive, not just nice, but clearly showing money spent in both clothing and accessories.
It's amazing how the greed of one man can cause so much damage. Ray Noorda is probably the biggest proponent of business development and investment and probably the direct cause of much of the employment in Utah Valley. I think is because instead of hoarding his significant earnings from Novell he has reinvested them and helped other people get dozens of businesses off the ground. This is especially true in the software sector but also in other areas since this is largely imported cash and exported services.
Without Ralph Yarro I wonder how much better the economy would be here, and in many other places around the country and to some extend the world... I wonder how many valuable resources that have gone other places would still be here and still be more involved as a proponent of open source... Like Ransom Love the ex-CEO of Caldera Systems, that purchased SCO and changed their name and direction soon after he left. In hindsight it makes me wonder if part of his reason for leaving was that he was opposed to such open and blatant attacks on Linux and open source software in general and didn't want to be a part of it.
Even more significantly, how much impact have the SCO Group's (what used to be Caldera Systems, a Linux and open source oriented company) actions had on other potential users of and contributors to open source software? Would such significant changes have come internally without investor influence? My guess is no.
The good news is that the SCO legal threat is diminishing. I must admit I don't trust our legal system to do what is best for people in general, that really isn't their job. They are their to judge and be involved in the enforcement of the law. This leads to some really really strange things happing in the legal world sometimes, that would seem totally illogical by normal logic, but in their application of laws to the current circumstances and in the name of merciless justice somehow to make sense and even seem obligatory.
Anyway, hopefully in this case the abuse of our legal system will not succeed. And who knows, now that hopefully the corruption at the top has been removed other powers that be will let go of the tantalizing taste of the temptation of power, wealth and comfort and just let the lawsuit go. Perhaps they will even re-join the ranks of those who are putting work into moving Linux forward for their own profit and for the progress of business and society in general. And perhaps they will get into pushing forward open source softare in other domains...
And what would be the benefit of this? Software is a unique resource that hasn't really been dealt with in history. It is cheap and easy to duplicate once initially developed and can result in significant savings and improvements in efficiency for those who use it. Not that it is free or easy to use in many applications where it is most valuable, but the tool alone enables increased efficiency that would otherwise be impossible.
The result of this in our modern economy is that massive corporations and monopolies have caused serious economic confusion and unprecedented shifts in control and distribution of wealth. That wealth in this world seems to be easier and easier to leverage almost directly into power. The danger of that should be pretty clear...
The furthering of open source software can help re-balance this and put control back into the hands of those who use the software rather than them being at the mercy of monopolies that enforce their position by taking advantage of their market share to make it hard for existing users to play well with users of other software.
Open standards are good, but the industry incumbents have to adopt them as well in order for them to succeed, which often isn't in the best interest of the incumbent.
I guess that's all for now. Will be an interesting show to watch as things progress...
I know The Open For Business Project has many shortcomings, I get dozens of email messages every week that tell me about them! Of course, other people may not be having the same problems as those people I hear from, there are a wide variety of needs and solutions to those needs. But that's okay because this isn't a company, it's a community!
If I could choose how to structure an open source project it would be this: spend the tens of thousands of dollars required to setup a tax-exempt not for profit company, and have millions of dollars of corporate and individual contributions to make a beautiful piece of software that everyone loves and can effortlessly use for free.
Oh, and all major contributors would be moved to a single beautiful place and have huge houses and drive Audi A8s or something similar. And of course there would be a staff of hundreds to keep up with documentation, support, business development, and so on so that the developers could sit around all day and pontificate and create wonderful things. And after major releases they would have time off since that would bring in serious financial contributions automatically by the generosity of corporations around the world so they would take a couple of months off and hang out with family and friends and lose some of the weight gained during development.
Is that too much a dream? Yeah, I guess it is... So, as slow (but really truly elegant and beautiful) as it is, here is how OFBiz REALLY runs. And actually, there are really big reasons why I love and you could too...
As far as scheduling an planning goes: this is a community driven project! Andy and I have no idea what your needs are and what you are working on unless you tell us. For us, we do work on some things on our own time (probably 30% of the project is in this category, and mostly in the framework), but we don't often know how much time we'll have to spend on those things. Most of what we do comes from contracts, so only the people who pay us know what is coming up. In many cases we don't even know until a few days or a few weeks before we get started. We never know which analysis and bid efforts will result in a contract.
In other words: this is an unfunded community project! Even worse, it is an unfunded community driven project that is still maturing! So, right now if you aren't interested in being part of the community with all that entails, I don't want you to go away at all, but I also can't afford to help you very much.
Maybe this little parable would be helpful:
Consider OFBiz as the multi-part project it is and that some parts are still in the phase of the growing wheat, and other parts (such as ecommerce) are already bread that some of us are getting fat from. In other words, there are still many opportunities in OFBiz to be a "Little Red Hen" and get all of the benefits that entails. I have no problem with ducks, cats, and dogs, but I also don't have the resources to make a way for you.
That said, we are certainly working on this all the time and after significant development periods a lot of documentation update is needed.
We will be doing some getting started guides for SPECIFIC functionality areas, or uses of OFBiz. It is almost impossible, and probably not terribly useful for beginners, to have a getting started guide that covers the entire project. So, the first one will be VERY ecommerce setup oriented and go over the _basic_ things needed to get an OFBiz instance configured for production.
We have a few contracts coming (hopefully) right now to do these sorts of basic ecommerce setups, so the timing is good. And, BTW, yes: this will include both development to simplify the process and documentation of what is left over. One OFBiz community member, Integral Business Solutions, has even offered to pay for some of this and to allow it to be distributed freely as part of the project.
I just want to keep one central message in this email:
This is a community project! Those who want to get involved do, in some way or other. Those who consistently contribute and act like real members of the inner circle become that. This includes people like Al Byers, Jacopo Cappellato, and Si Chen. It also includes Andy and me. And there are MANY others who contribute on a smaller scale what they can. Sometimes these are bug fixes or just bug reports. Sometimes these are helping new people with _correct_ and well researched and experienced information.
We are just members of a community. Some of us have chosen to be _intimately_ involved in this project and dedicate thousands of unpaid hours to moving it forward. This does require significant sacrifice and I understand that many people aren't in a position to put up this sort of time and money. I understand that because at this point _I_ could not afford to get something like OFBiz started again. Now, this isn't because consulting has been bad, I still make more than I did as a senior engineer and analyst, but I invest a lot of cash in equipment, paying other people, marketing, and so on and I've had a couple of really significant personal losses recently, one on the order of ~$25,000 and another around ~$90,000.
I'm sure many people reading this have experienced such things and I understand that sometimes things just don't work out or have to wait. For some people, using OFBiz is in this category: it hasn't progressed enough for them to able to _afford_ to use it. There are many of us working on that progression, but there is a lot of work to do. So, if it's not moving fast enough for you or is not convenient enough for you, there is only one person/company that can guarantee to fix that: you. Or, you can wait. It will get there, I don't think there's any question about that.
I appreciate that some people are taking the time to put some detail into this and describe their needs, instead of just saying "it sucks" or "it's crap" or "it's non-existent". Please understand that different people and different companies have VERY different needs so details are required to achieve any reasonable level of communication.
Thanks again to everyone who is working on this and constructively making effort to move it forward and help to solve problems that exist, and not doubt they do exist! Especially those who bring up and issue and propose a possible solution! That makes it much easier to discuss and understand, even if the initial proposed solution doesn't turn out to be very good!