(follow up to lilmatt’s blog post)
I want to write a bit on what Ian McKellar, my old fellow from Flock, and now a proud member of the Songbird team, called Not Invented Here attitude.
Actually, NIH syndrom is well known and described in the memoirs of Wikipedia. It’s a persistent sociological culture that prevents the organization from using existing knowledge, code or research because it has different origins.
While this topic has been widely discussed all around the globe, and even in open source world, it has never been a topic in a Mozilla ecosystem debate.
Pity, since there is an elephant in the room, I can swear.
Flock developers, Songbird developers, Miro, Joost, Disruptive Innovations (time to update the website Daniel), AllPeers, and developers from many other projects contributed to Mozilla source code in many different ways – look at the very rough bugzilla search that displays bugs that were reported or assigned to one of the developers from the projects listed above during last 2 years. Over 600 bugs! And only 22 of them are INVALID – pretty damn high signal/noise ratio. Those are much above average quality bugs, with higher chance to be valid, and fixed.
We have a lot of references to Flock on wiki.mozilla.org. Most of them refer to Flock’s UI on the UI draft list, which is obvious. Hey, we’re not talking about “stealing” ideas. We have two open source browsers that can exchange ideas, code etc. Flock is smaller, focused on people using web more than an average, it has a Bigger Brother, who has to move more carefully and be more responsible, while Flock can move faster experimenting with features from the new areas of the web usage. It’s obvious and natural that other browsers should look at what Flock is experimenting with. Same with Songbird. It’s an multimedia center that uses Gecko. If Gecko is about to add support for a media, the natural first look should be at how Songbird did that. I remember trying to get Termie to work on Places (it was before Fx 2.0 times! Ben Goodger was working on the Firefox UI), hey! Termie even was granted by his manager quite a lot hours per week to investigate and work on Places for Firefox. It was also an obvious decision from Flock. Flock will switch to Gecko 1.9. The better the places are, the easier it’ll be to use them.
So instead of Mozilla leveraging from this community, it often does nothing to use their knowledge, energy and code to make Mozilla better. At the same time, the companies around Mozilla are so focused on their own products, and are rarely doing anything more than minimum to contribute to Mozilla.
My best memorized case is bug 302387 from b.m.o. which is waiting for way too long lacking energy to push it through. On the other hand we have bug 2894 from Flock’s bugzilla, filed almost 2 years ago by Gerv. I can imagine how demotivated Gerv can be after waiting for so long, feeling that actually everyone is positive, but no one will take care to finish the procedure. There are many more bugs in both bugzilla’s, but of course majority is in b.m.o. Everyone is positive, no one complaints, but also no one does the work of the doorkeeper who helps an external guy make his patch through the system. (maybe I should get used to that, all in all, it’s exactly the same with MediaWiki for example). As you may expect, in the example above, Termie did not work on Places, and I believe the reason was because Flock did not encourage him too much, while Mozilla was not too much interested in getting help from a Flocker.
While it may seem like the guilt is divided, I believe that the stronger and bigger player keeps more responsibility for making the environment non-NIH.
So the first problem with this is that Mozilla community did not grew it’s doorkeepers. Wikipedia is a great example of a project where such people are grown naturally. In each and every Wikipedia community I tracked there are people who are specializing in greeting new ones and helping them skip the entry barrier. In case of Mozilla we should probably simply grow a culture of such attitude instead of separate group of people.
If we accept the fact that it’s really hard for an external person to understand and suit to Mozilla environment architecture and that once he did it (usually without much time granted by his manager) he faces extreme amount of passive slowdown. People are smiling, do the minimum they need when pointed, but nothing more, not a single step beyond. And in result I promise that without a deep sense of understanding, such person will never ever again want to go through this. He’ll just stick to he’s work.
Instead of the “Hey! Great you want to help us, let me help you” message, he gets “If you want to support us, prove that you have motivation to go through meanders of our policies, because we don’t trust the ones who can’t spend single 15 work hours looking for a super-reviewer.” – I understand that such policy raises the chance that the ones who actually will escape the pitfalls will contribute in a better way, but beside of that, it rejects a lot of valuable contribution. Too much comparing to what we win this way.
But even after fixing this we’re still not ready with the NIH problem. We’re still talking about the cases when a member of smaller Mozilla based project comes to the Mozilla bugzilla, files a bug and even spends some time to try to get it into the system.
The next step should be for the Mozilla to look around and accept the fact that in the open source world Mozilla built not only Firefox, not only Thunderbird, Seamonkey, Camino, Bugzilla and a community around itself. It also built many external projects, that are worth cooperating with. Not querying Songbird about audio/video, not querying Flock about starring, search bar, those are a cases when something went wrong. I’m not sure if that happened, but I can imagine that AllPeers members were not actively involved in download streaming upgrade for Gecko 1.9 (didn’t found any evidence of that in b.m.o). As Matthew stated, Flock has been never asked by Mozilla to tri-license any part of the code base. Even, when Firefox 3 “starring” system is strongly affected by the Flock approach.
Mozilla should get used to actively involving external project members into work on the topics which are affecting them. It requires a lot of groundwork, and is probably hard to do, but I believe it’s worth trying.
Let me state that it’s not about political issues. It’s about from-the-bottom attitude. It’s not happening always, but it happens often. It’s not a result of conscious effort, it’s rather a result of lack of effort that can be eventually strengthen by ambivalent reactions from managerial staff to the idea of using external support.
I can assume that NIH syndrome is a natural drift of most work environment. We have to put a pressure to avoid it, and to minimize the pressure, we have to be aware that we’re affected by this problem, plus agree that we all want to solve it.
17 replies on “not-invented-here syndrome in Mozilla”
You say “Flock has been never asked by “Mozilla” to tri-license any part of the code base” yet clearly that isn’t true. People have asked on several occasions to get small pieces of Flock code here and there and only now thanks to Matt pushing things through has anything come of it. The simple fact is that it is often easier to just build something new rather than try to get Flock to relicense it. Had Flock wanted to play nicely in the Mozilla world, they wouldn’t have used a separate license than everyone else. Trying to then push the relicensing burden on to others simply doesn’t make sense.
You say “Mozilla” doesn’t talk with people building things on top of it. Who is Mozilla here? Mozilla Corporation? Firefox/Gecko core contributors? It certainly isn’t clear from your post. As an individual contributor/module owner/etc I certainly talk with people from various other projects all the time. I’m not working on media support, but I’ve certainly chatted with Songbird folks about how they did various parts.
Over the last several years, I’ve seen a lot more people around the Mozilla project looking externally for help and reuse. Much more than ever before. Our use of cairo, ogg, jemalloc all come to mind.
You say “While it may seem like the guilt is divided, I believe that the stronger and bigger player keeps more responsibility for making the environment non-NIH.” — I disagree with this completely. Mozilla is built around openness and individual participation. If you want to make a difference or change things, show up and do it. Asking others to do it for you isn’t going to work.
Please make your blog posts viewable in completeness in an RSS feed please.
There was a lot of discussion of the HTML / API on the WHATWG mailing list. As far as I can recall, no-one from Joost or Songbird contributed to that. Chris posted patches, test builds, and regular updates on his progress on his blog and they were widely linked to. As far as I can recall, no-one from Joost or Songbird commented. I didn’t even know Songbird was doing video until today. How hard do you think Mozilla should work to solicit contributions and feedback?
Er, that was “the HTML <video>/<audio> API”.
We did actually look at the ZAP stack that Alex Fritze is using at Joost. It seems to be more complex than we need and very XPCOM-heavy so we decided not to go that way.
I reject the charge of “NIH”, at least in this case. Chris has imported a big pile of Ogg code, and worked with those guys to fix problems and get fixes upstream to Ogg. He even has Ogg commit access. He’s also using and contributing to the libsydney audio library. There’s no problem here with importing and using other people’s code.
Lowering the barriers to contributions is always going to be an issue. I try to do my part by helping new contributors as much as I can and trying to do consistently speedy reviews. I’m quite willing to be the “super-reviewer of last resort” when necessary! In the last few months we’ve had a sudden influx of contributors working on Linux theme issues which is very pleasing although it has created quite a bit of work for me … so it is definitely possible for new people to get involved without too much trouble. I would appreciate feedback on specific issues that we can improve on.
I have found as a new comer that people like reed help a lot to lower the barier of entry by asking reviews/approval on behalf of people like me, checking in for people who don’t have CVS access, explaining the rules a lot of times, etc. And I’m sure it takes him a good chunk of his time.
And that’s the real problem; I’m pretty sure “Mozillians” don’t do their worst to higher the ‘get a patch in’ barrier, it’s just that most of them don’t have time to help everybody (yet they are willing to inprove); that means that sometimes you’ll have to find for yourself who to ask superreview to, or sometimes you ask a review to someone who is alas completely overflown with reviews (and you don’t know it), etc.
@Stuart:
“People have asked on several occasions to get small pieces of Flock code here and there and only now thanks to Matt pushing things through has anything come of it.”
Can you give me an example of this? I was a strong advocate of upstreaming whatever/whenever possible and contribute to Mozilla. I know that Yosh, Lilmatt, Ian, Peter, Erwan were and are contributing to Mozilla. So if they were in charge of the piece you wanted to get upstreamed, I hardly believe it didn’t happen 🙁 So please, could you give me an example or two? I’d love to be able to analyze the cases.
Since I’m not in a good position to do that, I’d prefer not to discuss the licensing of Flock, since it’s not an issue. Flock is just a case. Songbird is also GPLed. If you prefer, imagine a browser X based on Gecko. Catch this – Mozilla policy *allows* to choose GPL as an only license for project based on Gecko. So it *may* happen. Do we always want to say “we don’t like you.” in each case? I do agree that it often might be just easier and faster to work on your own. But while the “faster” part makes total sense to me, the “easier” is just a mental trap that is a pure reason of NIH syndrome. It’s easier for me to write my own wiki system than fight with this MediaWiki. It’s easier to write my own gaming engine than fight with that freakin Crystal Space. I saw this tens of times. It may be easier, cause it requires the skills you have – coding, and excludes the unpleasant part of asking for help, asking for relicensing, asking for code, and learning someone else code. But it’s just the beginning. Once you pass such cycle a few times, people on both sides learn on how to play together and will make the procedures easier etc. It can hardly happen BEFORE cooperation begins.
“Who is Mozilla here?” – Mozilla ecosystem as it’s seen from outside. So in vast majority it’s probably MoCo employees & mozilla.org project members. There is no straight line, there is no guilt, I just see this is as a problem in our project. In Open Source you cannot say – you’re a member, and you’re not. There is a gradual line from the center of the project where the most important drivers are to the person on the local mozilla forum in Uganda who gives support to Fx users there. On this gradual line, there are tens of thousands of people and I don’t believe anyone will have courage to decide who’s “in Mozilla project” and who’s “outside”. On the other hard, we can say that something happens “in Mozilla project”, and everyone involved is a part of it. That’s all what I meant.
“I disagree with this completely. Mozilla is built around openness and individual participation. If you want to make a difference or change things, show up and do it. Asking others to do it for you isn’t going to work.”
That’s probably an example of blind man and an elephant (http://en.wikipedia.org/wiki/Blind_Men_and_an_Elephant) problem. I do agree with you and at the same time I keep my position that Mozilla has more power to grow a culture of teaching cooperation in its ecosystem so that the ones outside can learn. It’s a bit like a bunch of kids in a school, friends, team, pack. And then there’s this new boy. You can say “If he wants to join us, we’re open, he just need to follow our rules and routines and no one says ‘no'”, but that boy may have his own fears, internal trends to drift away from what you represent, voices inside him which’d prefer not to play according to the rules etc. If you can give him a hand, and help join the pack, the ones inside his mind who support this, will win, and he’s fears will be overturned. If you leave him on his own, it may not happen. It’s up to you, of course. You don’t NEED him in the pack that desperate, I agree. But synergy is a beautiful thing 🙂
@Havvy: I did it, but I’m not confident I want it. I prefer when planets show a summary, instead of “full lengthy posts” :/
@roc:
“How hard do you think Mozilla should work to solicit contributions and feedback?”
I believe it doesn’t work that way. I don’t think Mozilla should spend a lot of time to get this done. But building awareness of this problem, and building a kind of (oh, how can I say that…) empathy toward those projects which are around us may be helpful. It’s probably impossible to build a strong policy of who should contact who. I just believe that it’s not good when it doesn’t happen.
“I reject the charge of “NIH”, at least in this case.”
I’m not accusing you of that. The listed were just an examples to show how the problem might look like. It’s a micro/macro scale. People tend to think in a ‘micro’ – Did I ask for help? Did he help me? Ok, so I’m not guilty, phew… I believe that NIH exists in a macro world. You cannot accuse people of NIH because they didn’t cooperate on one element, one API, one bug. But you may say there’s a NIH problem when you spent 2,5 years in such an external project. And basing on my discussions with people from other projects, it just happens.
“so it is definitely possible for new people to get involved without too much trouble.”
Basing on what you said in your comment and what I know from Mozilla world, I’d even say that it’s easier for Mozilla to cooperate with external projects from outside of Mozilla scope (ogg, 7zip, gtk are the cases) than from within our island. It’s also not a case of problems with new contributors, I know people are helpful, I remember getting a lot of help at the beginning from Boris, Benjamin, Axel, you and others. But the case is that devs from external projects are not the same kind of people as newbies. We can both guess that statistically their input will be more specific, they’re more likely to write patches, to have some experience from other project (for good or bad – they may have strong habits from those), but it’s also important that they’ll have less motivation to pass the entry barrier test. They have their projects, and they want to contribute at the beginning just to play fair (that’s their point of view) or to reduce the amount of forking. So if they don’t see Mozilla’s hand helping them more than an average, usually highly motivated individual, they just easily give up and build an assumption that Mozilla doesn’t like them so it’s easier to keep their patches in their tree and focus on their product.
“I would appreciate feedback on specific issues that we can improve on.”
I believe that meeting together and letting them present what they’re working on to Mozillians may be really helpful. And as far as I know there’s an event in MtV of such kind going to happen pretty soon – good 🙂
Having worked as an outsider on both Mozilla and the kernel, I can say that it many times harder to get code into Mozilla than the kernel. I still think it is too hard to get code into both of them.
For example the kernel suffers from “port and forget” by embedded CPU makers. These manufacturers will pay for a port of Linux to their hardware. But it may take months or even years of wrangling to get these ports into the kernel, so the manufacturers have stopped trying since it isn’t cost effective. Linux is trying to fix this now.
Distributed version control (git) makes a huge difference in the flow of the review process. For example Linus is the top reviewer and he hardly writes any kernel code now. Other people at the top of the Linux food chain spend 50% of their time reviewing. I believe Mozilla is missing this non-coder reviewer element.
I’ve stopped writing code for Mozilla since it just sits in Bugzilla and never gets applied.
Point well made, Z.
I think it might not be a terrible time to have a regular Mozilla contributor or two set aside some amount of time for developer outreach and hand-holding. They would help contributors do “first bugs,” review patches for proper coding style so they’re ready for the actual review (which would ideally be done in a timely manner).
I’ve seen this happen in a few bugs, but I think explicitly dedicating time to the task and advertising the program would be very useful.
Gandalf, sadly, you post is mostly off-topic. NIH has nothing to do with the problems you describe.
All the stuff around the licensing is known, and is unfortunate. We’ll be stuck with the GPL for a while, and it’s a license that’s not good for end-user products, believe it or not. It might be good for developers, depending on your state of mind. Having to triple license Mozilla was unfortunate, it enabled things like songbird and other stuff with virally licensed libs, but it does make getting stuff back upstream hard.
So yes, finding out what it takes to land a patch is hard. Hell, I don’t know our reviewing rules these days. Which tells you that landing patches in Mozilla is hard. Not that it has something to do with “invented here” versus elsewhere. Note at the side, using RDF bugs as examples in this context just makes your argument lame. Not because it’s my module, but because you know that RDF will get nuked independent of what we do to it, so there’s no effort to improve it.
The biggest problem I have with your post, though, is that you make it sound like MoCo should care about other people shipping their products, instead of shipping its own.
And that, as utopian nice as it’d be, is just not the right thing to do.
That said, MDC is much better than what we had when you started, and I guess that anybody could just go ahead and invite shaver over for a lecture on how to contribute some company’s code back upstream, or just join a devday whish spread across the globe. There is the Evangelism team. Yeah, right, the Evangelism team, a whole devision at moco that only cares about our ecosystem.
So, while the actual problems you describe are there, and happen too often, your analysis doesn’t fit. It’s neither “NIH”, nor is it something MoCo isn’t working on.
@Axel:
“The biggest problem I have with your post, though, is that you make it sound like MoCo should care about other people shipping their products, instead of shipping its own.”
It’s pretty possible that you have a problem with it because I didn’t mean it. I believe that Mozilla should care about helping other people/organizations contribute to Mozilla, because their contribution can be extremely worth it.
Additionally, I believe that Mozilla should grew a separate model of treating external contributors who are a members of Mozilla based project differently from an external private individual volunteer who wants to start contributing. And the reason for that is because those two kinds of contributors are totally different. Driven by different motivators, satisfiers, pushes and pulls. That’s all what I meant. It’s not about other people products. (Although I disagree with you that it’s not in Mozilla business to make sure that the environment is broader than our products). It’s about what we do to let others contribute, and do we encourage them and help them, or just we make sure there are no major blockers. Two attitudes, two visions and two different results.
“Can you give me an example of this? I was a strong advocate of upstreaming whatever/whenever possible and contribute to Mozilla. I know that Yosh, Lilmatt, Ian, Peter, Erwan were and are contributing to Mozilla. So if they were in charge of the piece you wanted to get upstreamed, I hardly believe it didn’t happen 🙁 So please, could you give me an example or two? I’d love to be able to analyze the cases.”
Lilly and I Visited Flock several years ago in person to talk specifically about sharing code. MConnor and I personally visited Joost in their offices in the Netherlands after last Fosdem for exactly the same reasons. I’ve talked personally with the Songbird folks many times. I can continue to cite others. In all cases I believe the core issue is that these folks are busy trying to ship their own products and getting code committed back to Mozilla is *a lot* of work. We can and should make it easier for folks to contribute – but I reject the assertion that it is NIH on either side of the fence here.
Jon, code sitting in Bugzilla is a failure mode we want to eliminate, not the way we “want” things to be. In the past, we’ve had problems with unresponsive module owners; if we’re having that now, please raise the issue with drivers?
@Stuart
I too am curious what specific instances you have where Mozilla asked Flock to re-license something and that didn’t happen. The only request I’m aware of was for the tagging UI widgets, which is the recent one you referred to in your post. Since I’m not aware of any other requests, I’m wondering where communication broke down here, because any request I would have heard of I’d have tried to make happen.
Any change we make to base platform code we try to push upstream, whether they be bug fixes or enhancements. The only exceptions are some quick hacks that we want to refactor or get rid of anyway, nothing that would be acceptable upstream.
@Schrep
I think part of the problem here is that Mozilla doesn’t necessarily pay much attention what’s going on downstream, so this comes across as NIH. Given how much legwork it takes to even get comments on potential ideas in Bugzilla, it’s hard to justify time to push new features upstream, especially considering that Mozilla’s rather conservative attitude regarding adding core features would mean a high rejection rate.
So any changes to upstream code should be pushed upstream, that’s a no-brainer, and we at Flock certainly have been doing that. This has a clear benefit to both organizations. Contributing back wholly new features is another matter, and it’s unclear how to make this work smoother.
I’ve seen some of Jon Smirl’s patches in bugs, and I recall them getting in. So I checked:
http://tinyurl.com/2rmtzh
15 out of 74 are still NEW (none are UNCONFIRMED or ASSIGNED):
http://tinyurl.com/2jum9y
Some low-numbered bugs show ball-dropping by Netscapers, long gone. The bugs have not been touched in a while, generally. If any of these patches is still needed, then we need owners to act accordingly. I will follow up.
In more than a few of the more recent bugs, where the patch manager was around to help request review, I see a failure by Jon and others who might do it to set the r? flag to name a peer or owner, causing that person to get email.
Yes, the owner and peer should review incoming bugs and jump on any that get patches, where the r? flag wasn’t set (I and others do this for JS engine bugs; yay for us, as we non-NIH our way to Tamarin integration).
But mistakes happen, and the patch submitter has to be vigilant too. We probably don’t want bugzilla’s attachment UI to force or default r? flag setting.
On the general “NIH” claim, Mozilla is hardly immune but also far from convicted. We use other code, rewrite over time to get rid of our own old baggage, navel-gaze over Webkit, etc. etc. To use the “NIH” acronym carelessly, especially when downstreams are not only busy with their own products, but venture-funded and conflicted enough to use license poison pills — this careless charge (I think that’s what it is, I’m giving benefit of doubt — it could be something more like malicious) frankly stinks, IMHO.
Mozilla has code review requirements, and we aren’t going to drop those. We’ve added unit test requirements, sort of (reviewers and others require them or just know to do the work). We aren’t going to lower these barriers to entry.
Who is doing what, Mozilla jargon, how to work the system if you run into a lack of respose — these are all things to work on, and it’s great to see Reed credited with helping. IRC is important to our working culture, maybe too important, but if it ain’t broke… Anyway, this is not the first time we’ve had complaints about getting patches in, and as usual, when I dig into particulars, there are recurrent low-level problems on “Mozilla’s” side *and* on patch submitters.
Mostly these problems seem to be background loss of owner attention, sometimes due to silent loss of owner. I’m talking about the Mozilla side. Patch attachers may simply not know to set r? to an owner or peer bugzilla id. There’s probably no technological fix, but there could be better bugzilla UI.
Patching before collaborating on design is another problem. No project wants that cart before the design horse; it sucks to have to reset a complex, possibly useful (in part) patch to the design review stage it missed. Design review means using the newsgroups as well as IRC, the wiki, even blogs — and of course bugs, but with design docs before code patches.
I see more design collaboration over time, not less, in spite of Ian’s complaints, and I agree with roc that cdouble’s work on video was more than open enough for others not to throw NIH stones. Chris is in town this week, so if Songbird folk want to host something, or make the trip down to Mt. View, I bet he’d be up for it.
/be
I think 15 out of 74, or a bit over 20% unresolved, and the rest resolved one way or another (Jon, reopen if any were wrongly closed), is pretty good, by the way. Not great, but not as bad as you might think from Jon’s comment.
/be
[…] posted recently about Making good on a promise. zbraniecki followed up with a post on not-invented-here syndrome. I just wanted to […]
I’d be really disappointed if what comes out of this discussion is only finger-pointing and bitching, instead of more positive ideas.
Especially Bo idea about devoting more time to hand-holding people in bugzilla is IMO a great thing to do. I try to do it from time to time, sometimes it takes little more than one sentence, but I certainly only do it once in a while, and it would be great to emphasize that on a larger scale.