In its simplest form, a WCF Data Service can be built by exposing your ADO.NET Entity Framework ObjectContext through the DataService<T> class.
However, when it comes to testing, especially in situations when you have to enhance data in the OData Feed with some additional information (through WCF Behaviors for example), or when you have to verify that your Query Interceptors work as designed, using Object Context that is bound to a live database is hard because:

• Your tests that have little to do with the shape of your data become dependent on the database, which is a big problem
• You have maintain test data in the database when you run your tests, which is a maintenance overhead

So there lies our problem: how to expose data using WCF Data Services in a way where the database dependency can be mocked out during unit testing? The short answer is that we need to achieve better testability by introducing persistence-ignorant domain layer and by making our existing Object Context implement an interface that can be mocked/substituted in testing.

This article describes the concrete incremental steps you can take to get you from a standard Data Service built on top of Entity Framework Object Context to a solution that will satisfy testability requirements above. I will demonstrate the technique on a sample service connected to the AdventureWorksLT database.

Initial State: Data Service Of Object Context

public class MyDataService : DataService
{
    public static void InitializeService(DataServiceConfiguration config)
    {
        config.SetEntitySetAccessRule("Products", EntitySetRights.All);
        config.SetEntitySetAccessRule("ProductCategories", EntitySetRights.All);
        config.SetEntitySetAccessRule("ProductDescriptions", EntitySetRights.All);
        config.SetEntitySetAccessRule("ProductModelProductDescriptions", EntitySetRights.All);
        config.SetEntitySetAccessRule("ProductModels", EntitySetRights.All);
        config.SetEntitySetAccessRule("SalesOrderHeaders", EntitySetRights.All);
        config.SetEntitySetAccessRule("SalesOrderDetails", EntitySetRights.All);

        config.UseVerboseErrors = true;
        config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
    }
}

Step 1: Create an Interface that represents you Data Repository

This step is easy: For every entity type you expose in your Data Service, add a method that returns IQueryable<T>:

namespace MyDataService.DataAccess
{
    public interface IMyDataRepository
    {
        IQueryable</pre>
<address>Addresses { get; }
 IQueryable Customers { get; }
 IQueryable CustomerAddresses { get; }
 IQueryable Products { get; }
 IQueryable ProductCategories { get; }
 IQueryable ProductDescriptions { get; }
 IQueryable ProductModelProductDescriptions { get; }
 IQueryable ProductModels { get; }
 IQueryable SalesOrderHeaders { get; }
 IQueryable SalesOrderDetails { get; }
 }
}

After interface has been created, you can provide default/production implementation in the hand-written partial class that corresponds to your Object Context by just returning Object Sets already defined by you Object Context:

namespace MyDataService.DataAccess
{
    public partial class AdventureWorksLTEntities: IMyDataRepository
    {
        IQueryable IMyDataRepository.Products
        { get { return this.Products; } }

        IQueryable IMyDataRepository.ProductCategories
        { get { return this.ProductCategories; } }

        IQueryable IMyDataRepository.ProductDescriptions
        { get { return this.ProductDescriptions; } }

        IQueryable IMyDataRepository.ProductModelProductDescriptions
        { get { return this.ProductModelProductDescriptions; } }

        IQueryable IMyDataRepository.ProductModels
        { get { return this.ProductModels; } }

        IQueryable IMyDataRepository.SalesOrderHeaders
        { get { return this.SalesOrderHeaders; } }

        IQueryable IMyDataRepository.SalesOrderDetails
        { get { return this.SalesOrderDetails; } }

        IQueryable</pre>
<address>IMyDataRepository.Addresses
 { get { return this.Addresses; } }

 IQueryable IMyDataRepository.Customers
 { get { return this.Customers; } }

 IQueryable IMyDataRepository.CustomerAddresses
 { get { return this.CustomerAddresses; } }
 }
}

Now you have an interface in your system that defines data access methods already implemented by your Object Context.
This means that if you build the rest of the system to rely/depend on this interface, rather than use its implementation (i.e. the Object Context), you can achieve better flexibility and the testability without having to worry about test databases and policing test data in your build environment and on developers’ workstations.

This is a worthy and very achievable goal, but we have a few steps remaining to fully get us there.

Step 2: Change Code Generation Template to use Self-Tracking Entities

This step is needed because you typically do not want you entities to be database-bound, and instead you want a reusable, independent domain layer that can be used in multiple scenarios. This move will contribute to a better code flexibility and layering without jeopardizing data access functionality that Entity Framework provides out of the box. Full discussion on why separate persistence-ignorant domain layer is a good thing is out of scope for this article. If you want to research the subject further, I would start here, here and here.

Having done that, your Project containing Data Context will look differently:

• You will see AdventureWorks.Context.tt and code-generated Object Context in the AdventureWorks.Context.cs and AdventureWorks.Context.Extensions.cs
• You will see AdventureWorks.tt and code-generated entity classes that match you Entity Framework Model.

Note that entity classes do not have any dependency on the EntityObject and therefore do not have to live in your Data Access layer.

Step 3: Move Your POCO Entities into a separate “Domain” assembly

Now that we have a domain layer of self-tracking plain old CLR objects (POCOs), we want to make it reusable.
This means separating the entity classes into a separate assembly that does not have any dependencies on Entity Framework, System.Data or any other persistence technologies. The tricky part is that we want to still get the benefits of EF designer experience, i.e. if we added, removed or modified some entities, we want them to be re-generated the usual way.

To achieve that, follow these steps:

• Create a new Project where you intend to host your Domain Layer. Reference System.Runtime.Serialization.
• Right-Click on the Entity Code Generation template (AdventureWorks.tt) and select Exclude From Project.
• In your Domain Layer project, add existing Entity Code Generation Template (located in EDMX folder) as a link.Visual Studio will regenerate Domain Classes in the correct namespace that matches your Domain Layer:
• Reference your new Domain project from Data Layer and from the Web Project hosting your Data Service.
• Add using statements for your Domain namespace to fix compiler errors.
• You will notice that one of the places you had to add using statement is you code-generated AdventureWorks.Context.cs. Since this class is being generated every time, it is better to add additional using statement to the .Context.tt template:

  EntityContainer container = ItemCollection.GetItems().FirstOrDefault();
  if (container == null)
  {
      return "// No EntityContainer exists in the model, so no code was generated";
  }
  
  WriteHeader(fileManager, "MyDataService.Domain");
  BeginNamespace(namespaceName, code);
  

Step 4: Create your own Data Context that composes your Data Repository

In the Data Access project, add a class that will represent new context will use in the Data Service.
This class replaces Entity Context used before, and therefore it should implement all properties that return IQueryable instances for the Data Service. Since our Repository already implements all this, the implementation is trivial:

namespace MyDataService.DataAccess
{
    public class MyDataContext
    {
        public IMyDataRepository Repository { get; private set; }

        public MyDataContext(IMyDataRepository repository)
        {
            if (repository == null)
            {
                throw new ArgumentNullException("repository", "Expecting a valid Repository");
            }

            this.Repository = repository;
        }

        public MyDataContext(): this(new AdventureWorksLTEntities())
        {
        }

        public IQueryable</pre>
<address>Addresses
 {
 get { return this.Repository.Addresses; }
 }

 public IQueryable Customers
 {
 get { return this.Repository.Customers; }
 }

 public IQueryable CustomerAddresses
 {
 get { return this.Repository.CustomerAddresses; }
 }

 public IQueryable Products
 {
 get { return this.Repository.Products; }
 }

 public IQueryable SalesOrderDetails
 {
 get { return this.Repository.SalesOrderDetails; }
 }

 public IQueryable SalesOrderHeaders
 {
 get { return this.Repository.SalesOrderHeaders; }
 }

 public IQueryable ProductCategories
 {
 get { return this.Repository.ProductCategories; }
 }

 public IQueryable ProductDescriptions
 {
 get { return this.Repository.ProductDescriptions; }
 }

 public IQueryable ProductModelProductDescriptions
 {
 get { return this.Repository.ProductModelProductDescriptions; }
 }

 public IQueryable ProductModels
 {
 get { return this.Repository.ProductModels; }
 }
 }
}

Step 5: Update Data Service to use custom Data Context

[ServiceBehavior(IncludeExceptionDetailInFaults = true)]
public class MyDataService : DataService
{
    public static void InitializeService(DataServiceConfiguration config)
    {
        config.SetEntitySetAccessRule("Products", EntitySetRights.All);
        config.SetEntitySetAccessRule("ProductCategories", EntitySetRights.All);
        config.SetEntitySetAccessRule("ProductDescriptions", EntitySetRights.All);
        config.SetEntitySetAccessRule("ProductModelProductDescriptions", EntitySetRights.All);
        config.SetEntitySetAccessRule("ProductModels", EntitySetRights.All);
        config.SetEntitySetAccessRule("SalesOrderHeaders", EntitySetRights.All);
        config.SetEntitySetAccessRule("SalesOrderDetails", EntitySetRights.All);

        config.UseVerboseErrors = true;
        config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
    }
}

One you do that, Data Services infrastructure will switch from Entity Framework provider to a Reflection Provider.
There are subtle but important differences between the two. One of the symptoms of those differences is that Reflection Provider does not understand POCO’s the same way Entity Provider understands Entities.
If you run you service as is, you will get a bunch of errors stating that there are some unsupported properties.
To fix those errors, you would have to create partial classes for your Domain entities that do two things:

• Explicitly define data service keys
• Exclude Change Tracker from list of properties available for the Data Services to pick up

[DataServiceKey("AddressID")]
[IgnoreProperties("ChangeTracker")]
public partial class Address
{
}

Finally, verify that your service still works by testing it in the browser.
You should be able to perform any operation you did before, but now you can also Unit Test your service without having to have a live database to talk to!

Step 6: For testing, inject Mock Data Repository

To do that, we will need to figure out how to “legally” let Data Service know that we want it to use our implementation of the Data Repository
instead of default one that connects to a real database using Entity Framework. One way to do that is to use Inversion of Control and Dependency Injection. However, making Data Service and WCF aware of IoC Containers is a tricky proposition – there is a simpler solution

What if we created a factory method and used it to create instances of the custom context whenever the Data Service needs one? Fortunately, WCF Data Service provides a way to do that by overriding CreateDataSource method in the Data Service class:

public static Func DataContextFactory;

protected override MyDataContext CreateDataSource()
{
    return DataContextFactory != null ? DataContextFactory() : base.CreateDataSource();
}

Now our test code can set the Factory to create Data Context already initialized with a mock Repository.

This is, however, is only part of the problem – we also want to be able to host the Data Service inside of the test process without having to deploy anything to a real web server.
Once again, this is solvable by creating in-process service host:

public static class TestServiceHost
{
    internal static DataServiceHost Host
    {
        get;
        set;
    }

    internal static Uri HostEndpointUri
    {
        get
        {
            return Host.Description.Endpoints[0].Address.Uri;
        }
    }

    ///
<summary> /// Initialize WCF testing by starting services in-process
 /// </summary>
    public static void StartServices()
    {
        var baseServiceAddress = new Uri("http://localhost:8888");
        Host = new DataServiceHost(typeof(Web.MyDataService), new[] { baseServiceAddress });
        Host.AddServiceEndpoint(typeof(IRequestHandler), new WebHttpBinding(), "Service.svc");
        Host.Open();
        Trace.WriteLine(String.Format(
            "Started In-Process Services Host on Base Address {0}, Endpoint Address {1}",
            baseServiceAddress,
            HostEndpointUri));
    }

    public static void StopServices()
    {
        if (Host != null)
        {
            Host.Close();
            Trace.WriteLine(String.Format("Stopped In-Process Services Host on {0}", HostEndpointUri));
        }
    }
}

Now we can start writing tests that do not need the hosting environment and a database to verify that your service works as designed, especially if you start adjusting its behavior to add security, custom metadata extensions, query interceptors,
exception handling, etc.:

[TestClass]
public class DataServiceTest
{
    [TestInitialize]
    public void SetUp()
    {
        TestServiceHost.StartServices();
    }

    [TestCleanup]
    public void TearDown()
    {
        TestServiceHost.StopServices();
    }

    [TestMethod]
    public void DataService_QueryProductss_ReturnsCorrectData()
    {
        var expected = new List()
                            {
                                new Product { Name = "Product A" },
                                new Product { Name = "Product B" }
                            };
        var mockRepository = new Mock();
        mockRepository.Setup(r => r.Products).Returns(expected.AsQueryable());
        Web.MyDataService.DataContextFactory = () => new MyDataContext(mockRepository.Object);

        var target = new ServiceClient.MyDataContext(TestServiceHost.Host.Description.Endpoints[0].Address.Uri);
        var actual = target.Products.ToList();
        Assert.AreEqual(expected.Count(), actual.Count());
        Assert.IsTrue(actual.All(p => expected.Any(x => x.Name == p.Name)));
    }
}

Summary

Building applications that expose WCF Data Service is relatively easy. What is more difficult is to make these applications testable.

This article demonstrated how to do that by switching code generation template to self-tracking POCOs,
introducing persistence-ignorant domain layer and by making our existing Object Context implement an interface that can be mocked/substituted in testing.
Every one of these improvements is for the better, but curiously, all of them happened in response to one goal to
make our system more testable.

Feel free to explore Windows Azure Marketplace to find applications and data you can use for your business. Create your own app and become Windows Azure Marketplace Publisher!

Should we enforce mandatory code coverage standards by failing any build with low code coverage?

Well, I think that the answer to this question is a resounding NO, and here is why:

Reason 1: Coverage does not mean Quality

Low code coverage tells you that you don’t have enough tests. High code coverage tells you that you have a lot of tests. Neither tells you anything particularly useful about quality of your production code or, for that matter, quality of your tests. I readily concede that low code coverage is a smell of bad quality, but it’s not a proof.  Absence of unit tests is alarming, and should be corrected, but what if your production code is actually pretty good? On the other hand, high code coverage may be achieved by bad test code – imagine battery of unit tests with no asserts in them because what your code returns in unpredictable. Yes, your code coverage is high, but your production code stinks.  At best, code coverage is an indirect indicator of overall quality and therefore by no means should be considered a build blocker (something that randomizes your entire team).

Reason 2: You get exactly what you measure

If you evaluate your team success based on the code coverage trends, you will get the positive trends. You might, however, eventually land in a situation when you have good code coverage but bad quality, because when pushed to meet the code coverage bar (thing that you measure), people will cut quality corners. Compare this situation with measuring high-severity bugs that escaped your system all the way to the customer. That is a direct indicator of quality and something that is being done by an independent party (your customer), which is even better!

Reason 3: More process won’t solve people problems

So, we have a fact: Your engineers (notice that I bundle both Developers and Testers in one group) do not write enough unit tests to exercise most of your system. But why? What could be the real story behind this behavior?

There could be multiple scenarios at play here, but all of them point to People problems, not to lack of Process. Breaking low coverage builds constitutes nothing but more process and will not correct one of these possible people problems long-term:

• Our Engineers don’t understand the value of writing good tests.

Well, educate them! Explain that good test suites are not just burden, they are your safety net, your anti-regression armor, your documentation, you design driver, you early design verification and your sample code all in one!

• Our Engineers don’t have time to write tests – they are busy coding the features.

Your estimates don’t include enough time to outfit your features with proper test automation, and your management is apprehensive about spending 10 to 30% more time on each feature. I suggest a simple experiment: pick two features roughly similar in complexity, build one without tests and one with tests. Follow up by calculating number of bugs per feature that will follow in the next release and keep track of total amount of time, including shipping service packs and deploying quick fixes, that was needed in both cases. The results will be self-explanatory. Now you have hard data to take to your management.

• Our Developers don’t think that writing tests is in their job description.

Get another group of developers!

To summarize…

Measure your code coverage religiously and use it as one of the indicators of your overall quality. But don’t rely on it as your only indicator – don’t break your build over it, don’t yell at your engineers when code coverage drops 1%. Instead, make everybody aware why you think code coverage is important and review trends in your retrospective meetings. Encourage or even require your engineers to write tests to cover bugs and interesting usage scenarios, not your coverage targets. Enforce this rule using peer reviews. Correlate high-value bug escapes with components and use cases and adjust your efforts accordingly, so that if you have to improve things quickly, you can focus on unstable and buggy code first.

What is the environmental impact of the “Big Data“?

Let me explain…

Collectively, internet search companies have been relying on advertising dollars to make themselves profitable while keeping internet search free for consumers.  This, however, meant that compared “traditional” advertising models, search providers needed to make internet advertising much more economically viable for those companies paying for ads. In other words, they needed to improve ads targeting, which they did and are doing today very successfully.  But those models and algorithms, as scientifically amazing as they are, rely on massive amounts of data being collected about our site visits, our clicks, our searches, our behavior online.  All this data needs to be stored and processed somewhere. Indeed, the world is now running tens of millions of servers (estimated at 44 million). Significant portion of these servers (how big of a portion?) is dedicated to storing and processing search logs and clicks data directly related to online advertising. And all of those servers consume energy, a lot of energy (estimated at 0.5% of all electricity) and allegedly create significantly negative impact on the environment.

Yes, I know that data centers engineering and management science together with software and hardware manufactures made amazing progress when it comes to data center and software efficiency – but this is not the point of this post.

Regardless of how you feel about anthropogenic causes of global warming or about internet privacy, the fact is that we are still burning a lot (how much?) of energy… to do what exactly?

How much data do we need to index the web and make it helpful for all kinds of web queries? How fast does this portion of the data grow? How does it compare with volumes of data needed to continuously infer better and better ads targeting based on our behavior online and our social connections? In absolute terms per product sold, is environmental impact of online advertising better or worse than environmental impact of traditional advertising models? Can we get our hands on this statistical data? Is it a worthwile undertaking to understand it better?

In addition to some opinions out there about the questionable usefulness of online advertising, does online advertising basically helping us to “kill the planet”? Or is it as efficient environmentally as it seems to be economically? Are trends in data storage and processing sustainable with regards to the required energy production to support it? Should “Big Data” players bridge their considerable differences and collaborate on efficient and clean power production and distribution technologies with the same rigor they work on their own massively parallel data processing techniques?

I don’t know the answer to these questions – I am still searching for reliable data and useful models to estimate this.  If you have any suggestions – please feel free to reply to this post. I would be happy to collaborate!

• Developing cool new features before core functionality is in place is meaningless at best and distracting you from your mission at worst
• Developers left to their own devices tend to over-engineer most of the features and create flexible frameworks for everything, even trivial things
• Most developers suffer from NIH syndrome, which leads to everything being written at least twice if not more

Let’s assume now that the product is becoming successful. It is growing in size, gaining more and more fans and/or more and more revenue. At this stage blindly following Developer-Driven process can get you into even more trouble.

Business:

• New features developers chose to implement may not be helping your product
• New features developers chose to implement may not be what your users want most or want at all
• New features developers chose to implement may be cannibalizing or hurting other important revenue-generating features
• New features your developers chose to implement may not be the most profitable

Technical:

• Duplicated code grows all over the place and when new features are being introduced, raging debates start about what to use and what to retire and why. There is no consensus, so different factions continue to use and improve their code of choice ignoring others.
• Set of home-grown tools emerges that mostly work, but nobody is completely happy with them, so from time to time somebody creates new tools that do almost the same thing but “better”. Maintenance costs continue to grow…
• Documentation becomes so far outdated that knowledge about overall system architecture is priceless and can only be found with people who have been on the team since its inception.
• Build takes forever because of the amount of code. Configuration is mind-boggling because it is hard to configure dozens of “data-driven” frameworks to actually do what users want.

I could continue this list for a while, you know…

So what’s the solution? In my mind, it is very simple – provide a system of lightweight checks and balances that will keep the good and eliminate the bad:

1. Get a good PM on the team. Give them broad authority to influence and little authority to dictate. Make them work together with Dev and Test teams. Good PMs hate process just as much as Developers, and they will provide an excellent balance against cowboy development and over-engineering problems. More importantly, this move will compliment technology focus of Dev-driven product with razor-sharp customer focus.
2. Get architects to actually do their job. Architects who design how features should work should also be capable enough to quickly code up reusable and complicated components and provide actionable feedback and oversight to other team members. However, architects who sit in an ivory tower are just as bad for the product as no architecture at all. Therefore, getting seasoned developers to become architects (i.e. experts) on the product is a better way to go, I think.
3. Use experimentation in addition to (or instead of) traditional marketing research. Once a product get beyond initial rollout stages and starts gaining steam, the only way to grow effectively is to use data, quickly try out new features, measure, and if ideas fail (most of them do, by the way) – fail early and get out.
4. Watch over the build – avoid having dedicated build team, keep number of branches to a minimum, make developers responsible for integrating code. This will serve as a forcing function to have a reasonable component architecture and inject enough engineering discipline.
5. Do not subscribe to “good developers write bug free code” myth, invest in test infrastructure, measure test coverage. Once again, placing test responsibility with Developers can be used as a forcing function to inject stability and reason into your code base.

In short, maintain user-centric focus, keep it simple as much as possible, use real data, trust but verify.

Unfortunately, attempting to reconnect to another Team Project using “Configure Server Connection” option in the Team Foundation Ribbon will not work because it requires the same project collection and it primarily designed to work around backup/restore scenarios:

We will have to apply a workaround solution, which is the following:

1. Save a copy of original Excel Workbook and close it. Disable Team Foundation Add-In. Close Excel.

To disable TFS Add-in, go to File->Options->Add-Ins. Select COM Add-Ins in the Manage drop down and click GO:

In the dialog that shows up, uncheck Team Foundation Add-In and click OK:

2. Open the workbook and remove all custom document properties from the Workbook. These properties contain TFS binding information TFS Add-in will automatically recreate them when reconnecting to another Team Project. Not disabling Team Found add-in seems to allow it to re-create these properties even when saving the document.

This is a bit tricky and tedious, and you might want to use VBA for this. Click Alt+ F11, in the VBA Editor enter, and execute the following Macro:

Sub DeleteCustomProperties()

    Dim p As DocumentProperty

    For Each p In ActiveWorkbook.CustomDocumentProperties

        p.Delete()

    Next p

End Sub

3. Remove hidden verification worksheet. Save and close the Workbook.

This is also done in Visual Basic Editor because that page is hidden from the view. In the Project View in the Visual Basic, select a worksheet with name starting from “VSTS_ValidationWS” and in the Properties Editor change Visible property to xlSheetVisible:

After that, delete VSTS Validation Worksheet in Excel:

4. Re-enable TFS Add-in using the process opposite to the one described above. Open the Workbook again. Notice that when you navigate to the worksheet containing TFS-bound table regions, Team Ribbon does not activate Refresh button because it no longer knows that your workbook in connected to the TFS:

5. Make sure you do not delete the TFS-bound regions completely because if you do, your formulas will be damaged beyond repair. Keep at least one line from the old table and make sure you copy the name of the table to a text document somewhere – we will need it later:

6. Using New List button located in Team Ribbon, add another TFS-bound list that is connected to the same query in the destination Team Project and make sure column layout matches existing list. You will end up with two similar looking lists – one (with one remaining row) from old Team Project and one from the new Team Project:

7. Now use the old table name and new table name to adjust all formulas. Verify that formulas work as designed.

One again, this might be easier to accomplish with Visual Basic. Knowing old table name from step 5 and new table name from step 6, run the following macro (it might take a while to execute):

Sub ReplaceFormulas(ByVal originalListName As String,_
                    ByVal newListName As String)

    On Error Resume Next

    Dim w As Worksheet, i As Long, j As Long, _

        formulaValue As String

    For Each w In ActiveWorkbook.Worksheets

        For i = 1 To w.UsedRange.rows.Count

            For j = 1 To w.UsedRange.Columns.Count

                formulaValue = w.Cells(i, j).formula

                If formulaValue <> "" Then

                    formulaValue = Replace(formulaValue, _
                     originalListName, newListName, , , vbTextCompare)

                   w.Cells(i, j).formula = formulaValue

                End If

            Next j

        Next i

     Next w

End Sub

Sub ReplaceAll()

    Dim calc As XlCalculation
    Application.ScreenUpdating = False

    calc = Application.Calculation

    Application.Calculation = xlCalculationManual
    ReplaceFormulas("VSTS_8eeef3b2_74af_457d_88e8_0fce8bc5d131", _
                    "VSTS_d3ffaeb7_2a52_4d3e_afd8_2d7334bb47bb")
    Application.ScreenUpdating = True

    Application.Calculation = calc

End Sub

8. Remove old tables. The easiest way to accomplish this is to first convert it to a Range and then by deleting rows from the Worksheet:

9. Save modified Excel Workbook.

Note: If you have Pivot Table objects pointing to the old tables, you may need to augment ReplaceFormulas macro with updates to PivotTable objects – their Source Data needs to be updated.

Hope that helps!

So I started thinking about why this happens in this age of cutthroat competition. I don’t think it is Karma or “Pay it Forward“, though for some people out there that is a legitimate and commendable driver for this behavior. Instead, I think that it is a basic drive to look for and find better social interaction in our professional lives. Plus a lot of networking, of course. I firmly believe based on my own experiences that we strive to be accepted professionally, and these loosely coupled peer-to-peer communities satisfy that need. People treat each other as equals, assume the best instead of the worst, listen, share their advice and help because they like to do it. It gives them a great feeling of self-worth, especially when they get the same kind of treatment from the others.  Notice that this is slightly (and in some cases drastically) different from the corporate world. First, no matter how senior your position is within your company, you are constrained by its reporting structure and by its culture. In some situations you are treated as a resource, and in some situations your opinion or ideas have hard time reaching the right audience. In these startup communities, however, things are different because these constraints don’t exist. So meeting somebody for coffee and chatting with them about your and their ideas is a much simpler proposition.

So, in light of these observations I’ve made a resolution this year – to help my friends in their crazy ideas when they need my help, or advise, or a sounding board, or all three. Just to have a professional environment outside of work that is slightly different from what I have at work. Just to geek out with my buddies on my free time while feeling important. Just to be free for a few hours every week. Ha!

First, I know that uncertainty is inevitable and it is basically good. Without uncertainty, there won’t be any room for initiative, innovation, and risk taking.  However, too much uncertainty typically results from other bad symptoms that bring enough distraction to make me worry and question the project’s validity.  I’ve been trying to classify these most telling symptoms, and find ways to treat them. I am purposely staying away from management styles, social dynamics, reporting hierarchies on a project or even the project architecture, though these can have plenty of problems as well.  I am thinking about basic, fundamental issues that interfere with way to approach all technology projects: find what the problem is and proceed to create a technology that will solve it.

Here is what I could come up with, and I’ve been:

The NASA Pen Project

This a project where everybody assumes that problem at hand can only be solved with a new technology and diligently and stubbornly tries to come up with such a solution. Anybody who dares to suggest that the problem can be solved by using an existing technology (potentially mixed with some skillful organization changes) is immediately dismissed because… Nobody can tell why.  This creates a situation when people who are actually doing the work can’t fathom why or how their work might be useful because they all know that existing technology is just as good or even better. This uncertainty over the project’s fate causes quality issues, not to mention the team morale, because people think that their time and their work is being wasted. The only effective way I found to resolve this is to bring data to compare the projected costs of developing and implementing new technology with likely much lower costs of using existing tools to solve the problem and to propose a set of organization changes that will remove existing barriers.

The New Bicycle Project

This is classic and rampant NIH symptom. Because of political pressure within an organization or because of artificial competition and meaningless disagreements between “big wigs” in multiple divisions, an emerging or even an existing technology is reinvented with various degrees of success over and over and over again. Everybody has fun working on the cool new thing on their own, but company resources needed to solve the same problem are doubled or tripled. Nothing or very little is shared, teams hate each other, team management tries to outsmart their peers in the eyes of executives, and in the end nobody achieves the level of success they could have achieved had they worked together. Two-pronged attack might help here, I think: relentless pressure from the bottom (people demanding increased reuse and sharing of products already under way) and smart management from the top (timely executive decisions to combine or eliminate duplicate efforts).

The Emperor’s New Clothes Project

This is hysterical to observe from the outside but it is very painful for the people involved. This is a project where everybody seems to know and understand what the real problem is, but instead of solving it directly, team seems to be busy creating technology to work around it. After a few of those, a technological monstrosity/hodge-podge emerges around the original problem at hand, and it is usually unmanageable, unmaintainable, does not make things better at all, and is very costly. Once again, this could be due to politics within an organization or due to entrenched and seemingly unsolvable nature of the main problem (which sometimes is not technological at all). But until somebody has the guts to become that child who yelled that the emperor is naked, things will continue to be very difficult and the original problem will become even more entrenched because now there is a large investment into this pseudo-solution.

Go I Know Not Whither and Fetch I Know Not What Project

This is uncertainty squared – nobody seems to know what the problem is or was to begin with, nobody seems to understand why an existing solution, if any, is inadequate, and nobody can describe in concrete terms their vision for the new solution. But the plans are already approved, money allocated, and your team is on the hook to deliver… something. The solution likely lies in peeling the onion and asking a lot of questions from a lot of people. Start challenging every assumption to see how sure people are that there is a problem. Find out its exact nature to avoid building a space pen. Make sure you reuse existing bicycles. Call the emperor naked if you have to. Do whatever you have to to define “whither” and “what”.  And if you can’t make any headway, be aware that there is a chance that you got this fairy tale project because somebody wanted to remove you from the picture for a while or permanently…

Am I being too negative? Well, I don’t think so – I am an optimist. This is why I am doing my very best to prevent my new project from becoming one of these and I am beginning to like this challenge more and more.

Happy New Year!

Within every large self-respectful company, there is an R&D organization that targets long-term aspirations, works on crazy projects that may or may not pan out, makes lots of mistakes and sucks up a lot of company money. Executives accept that fact because entrenched R&D organizations provide future foundation for the company by doing both fundamental and applied research that is relevant to the company’s aspirations, even if in the short-term they have nothing to do with the problems at hand and the only thing that a company gets from their R&D folks next quarter is most likely a huge bill. I am not going to argue with this status quo because I think that just like fundamental scientific research done at universities, corporate R&D is basically a good thing.

What I wanted to talk about is so-called “Labs” projects that spring up from time to time in addition to primary research organizations.  Those guys act like startups – they are typically small driven teams that have a brilliant idea (or so they think at least) and that are lucky enough to have at least one executive who is willing to support them initially and let them work on that idea while protected them and supporting them by the money from corporate coffins. Sooner or later, however, this good will always ends. Always. Why? Because the fundamental goal of every corporation is to make profit and as far as crazy research goes, they already have that going on in their R&D department.  I would estimate this grace period at maybe 2 or 3 years. After that, a Lab/Startup project has to show a real profit and have a significant impact on the business to make its executive sponsor look like a rock star.

If not, 9 times out of 10 it is going to be axed, reorganized or split, I guarantee you.  More often than not, Labs don’t work because they make one of the following mistakes:

• They focus too much on internal needs first, provide “white glove treatment” to some of their internal customers and partners, neglecting other “less important” clients and choosing to not worry about basic viability and profitability of their product for the outside world.
• Conversely, they focus too much on external customers, trying to generate industry buzz, giving out glorious interviews and writing superb articles in industry publications, but they never try to make their product accepted and used internally.
• They neglect to formulate a long-term road map outfitted with both quality business model and clear technological improvements.
• Their mission includes grandiose aspirations to change internal corporate culture and deliver the world peace and forgets primal need to make gazillion dollars within the first 5 years of existence.
• Their internal structure resembles a garage project full of geeks having way too much fun for way too long, forgetting about releasing as often as they can and holding people accountable to the scope and schedule commitments they make.
• They get comfortable being shielded by their executive sponsor and forget about the fact that this executive is the only protection they have from packs of hungry wolves that have to deal with today’s problems and who bring home profit every quarter doing things that may be not so cutting edge.

So how do we create a sustainable internal startup that survives beyond the 2-3 years? By not doing these things. By making sure our strategy focuses on inside and outside needs. By working on business aspects of our idea as much as we work on technology.  By making our technology scale to our (prospective or real) customer needs early and well. By partnering with external organizations with which our company already has good relationships and asking, begging them to use our product and see if they like it. By making our executive sponsor look good every quarter because we landed another customer, or drastically increased scalability and performance, or shaved a significant chunk off our internal support costs, or shipped something really useful to our partners that saved or earned them real money.  By holding our development team’s feet to the fire all the time. By making our architects code and by making them help our developers understand their vision instead of isolating themselves into “I had a brilliant idea. I am great. Now you do the work” tower.  By constantly checking and challenging every decision our management makes against our long-term survival and profitability goals and against main goal of publicly releasing our product. By servicing as many customers as we can, even when it seems difficult. By relentlessly and obsessively eliminating every inefficiency and manual work in the product and in the process. By not expanding the headcount unless we have shown enough profit to pay for additional employees. By assuming that this quarter might be our last.

In short – by being a bunch of hungry artists.

Well, I am going to attempt to make an analogy between Project Management constraints and your Happiness at Work constraints. When I say “happiness at work”, I mean a situation when you are mostly happy to go to work every morning.  I don’t have any illusions that things could be perfect, at least not when you’ve been working in your current role for some time. I realize that there are always going to be problems, but those problems shouldn’t give you heartburn every second of every day to make you leave – those are the problems you or somebody else on your team is methodically working to solve and the rest of you are either helping or at least not standing in the way.

So what’s the constraints?  During my time in software development, I found that my personal happiness at work rests on three main constraints that are at the base of my professional Maslow’s Pyramid:

• Product I am building
• Team I am working with
• Technology I am using

When all three are balanced (once again, they can’t be all perfect all the time) – I am happy. When one is persistently screwed up, I am beginning to worry. When two are out of sorts, I am headed for the exit, unless I see a way to fix the problem. Let me explain why.

Product I am building gives me and everyone around me purpose. Multiple factors contribute to good or bad feelings about the product. I like when I believe in the product, when I think it solves a real problem, when I find the problem space fascinating, when I feel connected to my users (this could be just an illusion, right?). Take this all away, and suddenly there is no purpose, no goal, no mountain to climb. It could be OK for a short while, but it gets on my nerves very quickly.

Team I work with defines my social environment at work. I like to work in a team based on mutual respect, accountability, driven by results and not too political, and who doesn’t?  I also tend to pay attention to processes that exist in a team. How do ideas get communicated up the chain? Are they dismissed out of hand, do you have to be a member of  ”in” crowd to be heard? Is your management effective, accessible and open or are they just along for the ride? Do they listen to valid concerns and take action to make the team better? Do they defend the team or just themselves in battles that are worth fighting? Basically, I try to evaluate my team using one main criteria: can we conquer team dysfunction or not?

Technology I deal with on a daily basis is hugely important for me as well. Basically, I want to be able to answer “yes” when somebody asks me question #9 on the Joel Test. I have to spend my working days making technology do what I want, so it is only reasonable that I should be a pretty demanding customer. I don’t want the technology to waste my time – I want it to help me. I also want to enjoy working with it. I also want it to be marketable enough for me to learn. No matter how enjoyable and effective Smalltalk is, the reality check tells me that becoming a Smalltalk expert is probably not as good for my resume as knowing Java or C#. And this is coming from a person that loves Smalltalk. Same goes for internal tools that most companies have.  None of these tools are marketable (they are internal), so if they are clunky, old and hard to work with, I will hate them.

So what do I tend to do when any of these pillars becomes jeopardized? Well, it’s always the same two choices when it comes to your basic needs being threatened: fight or flight, except I alway fight first. I stay loyal, I try to influence non-technical decisions about the product, I try to lend my time, expertise and advise to fix my dysfunctional team or I try to push for technological choices that I consider more promising, applicable and marketable.

However, if I can’t convince myself that at least two out of three are acceptable to me, there is nothing I can do to make them acceptable and the situation isn’t getting better, I think that it’s time to move on. As loyal as I am, there is always another team, another project, another challenge.

Because life is too short to spend it in a dead-end job you hate.