Inefficiencies of large systems

I’ve spent most of my career in huge organizations. The overhead required to operate a large corporation is a business upon itself. The larger the team, the larger the effort to coordinate the efforts within and between organizational units. When an organization talks about flattening the org chart, it’s a result of out of control overhead. Computers systems are not immune from this impact. 

We see to impact of large organizations in our customer services experiences. After learning of a new service via a television commercial, we may call our cable TV provider to inquire about said service – anyone say, “Xfinity Gigabit” service? We soon find out that the customer service agents know nothing about the new offering. We find ourselves as customers educating the sales teams on the offerings of their organization. It’s a life hack as a result of hyper-convergence of companies. 

We see similar occurrences in large computer systems. It occurs in large ecosystems created by the same corporation. There are no two services more native to Microsoft than Windows Server and SQL Server. Both platforms are critical to the success of Microsoft. However, there’s still the need to create natural hacks to obtain the best performance between the two systems. 

Software company DataCore created such a hack in the form of an I/O filter. DataCore’s software installs as an I/O filter in Windows Server. As requests come from higher-level services in the user space such as SQL, DataCore uses proprietary algorithms to create parallel streams to the storage sub-system. The result is more bandwidth for the application I/O. 

The optimization isn’t something that the Windows Server team and the SQL Server team couldn’t have done together. However, Microsoft is a massive organization. The overhead to running the group creates inefficiencies that are bypassed via other parts of the ecosystem. In this case, DataCore’s offering. 

I discovered DataCore as a Tech Field Day 15 delegate. I wrote a preview of their presentation and recorded a recap of the session.