S3 logs to Statistical Reports

[samof76]

Cloud is the new buzzword in technology. Though the Cloud primarily meant the internet in many design or architectural diagrams, Cloud Computing has slowly but surely evolved into three well defined services, IaaS, PaaS and SaaS.

When we talk about IaaS; the front runner in this Service Space is Amazon with the service acronym AWS (Amazon Web Services). One of the service provided, and is widely accepted, is called Simple Storage Service(S3).

S3 as the expansion suggests is AWS' storage wing. S3 as of Feb 2009, stores 40 billion objects, no small number by any means. And AWS itself boasts of 490k developers in its community. So computing equal share we have about 82000 per developer. And all this translates into more dollars for Amazon and lesser dollars for the users. But with a pay as you go model, you would not really feel the pinch.

S3 is a very effective storage mechanism for whoever wants many a bytes in the Cloud. All you need is an AWS account, and you are ready to go by creating S3-wide unique objects called buckets. Bucket names generally indicate the nature of the contents. For example, if your company is called "Open Ink" and you would want store your Giga/Tera bytes of inventory data on the S3 you could create a bucket "openink-inventory", and move all your data there. AWS charges for the bandwidth, and the storage, you have used.

Since you might not be the only person in your company to access this data. And each access means a drop dries out of your pocket. You might need a mechanism to somehow track what is going on, who is accessing, where is it accessed from, etc.

Sadly, the only way you could track is by enabling bucket logging, and then reading through those cryptic logs, to say the least. But those are effective logs, none the less, and they prove to be helpful data to a handful of tools which could give effective reports. I will look at three tools, Sisense, S3Stat, CloudBuddy Analytics, and see how they compare.

                Sisense      S3Stat       CloudBuddy Analytics
Basic
 
Size               11M          -             9M
Platform           Windows      -             Windows
MultiAccount       Yes          No            Yes  
MultiUser          No           No            Yes
Engine             Prism        Webalizer     AWStats
Install            Local        -             Local(Web Based)
Cost               Trial        $5/month      Free(Open Source)
 
Reports
 
Month-wise        Yes            Yes          Yes
Weekly            Yes            Yes          Yes
Hourly            Yes            Yes          Yes
Bandwidth         Yes            Yes          Yes
OS type           Yes            Yes          Yes
File type         Yes            Yes          Yes
IP Geography      Yes            Yes          Yes
Top Files         Yes            Yes          Yes
Visits            Yes            Yes          Yes
Browsers          Yes            Yes          Yes
Customizable      Yes            No           Yes*
Extensible        No             No           Yes
 
Others
 
UI                Good           Basic        Basic
Usage Ease        Cumbersome     Easy         Easy
 
* - You need to drill the code.
 
 

Looking at the feature set they all sell the same thing packaged differently. So its very hard to jump to any kind of conclusion on the features they provide. But on the other fronts like Size, Costing, UI, and Extensibility there are many differences upon which you could make you choice.

References:
http://www.datacenterknowledge.com/a...llion-objects/
http://aws.amazon.com/s3/
http://www.mycloudbuddy.com/
http://www.s3stat.com/
http://www.sisense.com/AmazonS3.aspx