Typical “Big” Data Architecture

Here is the typical “Big” data architecture, that covers most components involved in the data pipeline. More or less, we have the same architecture in production in number of places (with some varying components due to data sources and data consumption is varied from company to company).

“Big” Data Architecture

Any data architecture loosely consists of four major logical components:

1. Data Source

True source of data coming from heterogeneous data sources. This is typically your data stores (SQL or NoSQL) that gives a structured data or any other data coming through APIs or other means (semi-structured or un-structured).

• Data from SQL, NoSQL stores (MySQL, Oracle, PostgreSQL, MongoDB, etc. – Mostly structured)
• (Semi/Un)-structured data (CRM, marketing, campaign, spend, revenue, leads,  etc.)
• Web logs or other log files (weblogs, user clicks, user visits, activity, etc.)

2. Data Transformation

Transformation of data from one form to another, its either part of ETL (Extract, Transform and Load) or import/export tools and/or scripts. Mainly used to load all sources of data into data processing pipeline.

Log management tools can also be considered as part of ETL, as they generate useful events from log files and present dashboard with alerting system in place or they can be directly loaded into data processing stores.

• ETL, ELTL tools (bash/python/perl/java scripts, Business Objects, SSIS, Kettle, etc.)
• SQOOP (Data Source to Hadoop data transformation tool, JDBC compatible)
• Import/Export tool (SQL/NoSQL vendor specific tools)
• Log Management tools (Splunk, Syslog, Custom log filter scripts, flume, scribe, loggly, etc.)

3. Data Processing or Data Integration

This is yet another source of vast data by combining both structured and un-structured data in one place (either real-time or incremental loading); mainly for data processing (Data Warehousing or Analytics) and generate usable data (materialized or aggregated) that can be consumed by data consumption components.

• Hadoop and Ecosystem (Hadoop/HDFS, Map-reduce, HBase, Hive, Impala, Pig etc) – uses HDFS as native storage
• Data Warehouse and Analytics solution (MySQL, SQL Server, Vertica, Green Plum, Aster Data, Exadata, SAP HANA, IBM Netezza, IBM Pure Data, Tera Data, etc.) – Uses vendor specific storage, optionally uses HDFS, even though with degraded performance.
• In-memory Analytics (SAS, Kognitio, Druid, etc.). This is an emerging market and trying to take advantage by reading directly from HDFS. We will see lot of in-memory analytics in coming days.

4. Data Consumption

Data consumption components that either consumes or exposes the data in usable form to end users or to other layers internally (ad-hoc) or externally (using APIs)

• Reporting (custom dashboards, micro strategy, pentaho, business objects, cognos, hyperion, tableau,  etc.)
• Search or Discovery (solr, elastic search, tibco spotfire, datameer etc.)
• Data Science, Mining and Analysis (mainly for internal data analysis to predict or estimate the overall performance and also drive recommendation using set of algorithms, user defined map-reduce jobs or ad-hoc queries)

Apart from the four logical components, monitoring plays a crucial role in detecting any failure within the data pipeline along with threshold changes to identify any bottlenecks in terms of performance, scalability and overall throughput.

• 
•