Friday, June 28, 2013

Custom hardware/software and scale

Though this is a start, I think at scale, the answer to all the problems is that you can not solve high scale issues with consumer level solutions.  So ironically a thing that started with using off the shelf consumer grade hardware and software as a base and built custom solutions on top of it, again needs specialization. Or the other point is that if there is one flaw in this suggested approach, it is the fact that this approach is not universal.
Most of the technologies in this area of high scalability do use standard software and hard ware at its base and build on top of it.
Every thing including big table,big data and other associated technologies do use the off-the-shelf file system and off-the-shelf operating system on individual nodes. The speed and scalability is achieved by software running on these machines by custom software which most of the times runs in user space as well as is written in high level languages (mostly JAVA).
So even when the suggested approach works, it is definitely needed to first work for the individual machines for existing software stack in consumer space before it gets really prevalent usage.