Jonathan Wilson https://jonathanwil.com/blog/tech.html quarto-1.7.32 Sun, 12 Jul 2026 00:00:00 GMT Understanding the Hadoop Ecosystem: A Concept Map for Big Data Architecture Jonathan Wilson https://jonathanwil.com/blog/posts/tech/hadoop/hadoop-ecosystem-paper.html

Introduction

Big Data and Why Hadoop

In 2006, British mathematician Clive Humby coined the phrase, “Data is the new oil,” and today it seems more relevant than ever (University of Sheffield, n.d.). In 2002, about 5 exabytes of new information were stored worldwide, while IDC projected that the global datasphere would reach 175 zettabytes by 2025 (Lyman & Varian, 2003; Reinsel et al., 2018). If data is the oil, then what are the refineries? What systems can process these enormous amounts of data and turn them into useful products?

Organizations need ways to store and process data beyond the limits of a single server and many traditional databases. Instead of relying on one increasingly powerful machine, distributed systems divide the work across many computers. As Grace Hopper explained through her oxen analogy, when one ox cannot move a heavy load, you add more oxen rather than trying to build a larger one (Hopper, 1982). Hadoop follows this same idea by allowing clusters of computers to store and process data together.

What is Hadoop

Apache Hadoop is best understood as an open-source big data ecosystem rather than a single tool. It provides a framework for storing and processing large datasets across clusters of computers. Its four core modules—Hadoop Common, HDFS, YARN, and MapReduce—support distributed storage, resource management, and parallel processing, while related projects extend Hadoop into areas such as querying, analytics, governance, and data management. Unlike a traditional database, Hadoop is designed to scale from a single machine to thousands of computers working together (Apache Hadoop Project, n.d.).

Origins of Hadoop

Hadoop was born from two key developments that happened around the same time in the early 2000s, when web search and large-scale data processing were becoming increasingly difficult to manage with traditional systems. One major development was Google’s work on web-scale search. PageRank, created by Larry Page and Sergey Brin, helped make Google successful by ranking web pages based on link relationships across the web (Brin & Page, 1998). As Google grew, it also needed a better way to process massive amounts of web data across many machines, which led Jeffrey Dean and Sanjay Ghemawat to publish the influential MapReduce paper (Dean & Ghemawat, 2004). Around the same time, Doug Cutting, the creator of Apache Lucene, was working on Apache Nutch, an open-source web search project that needed scalable storage and processing. Hadoop grew out of this work as part of the Lucene/Nutch ecosystem, and its interesting name famously came from Cutting’s child’s stuffed toy elephant. The combination of Google’s MapReduce ideas, the needs of open-source web search, and the work being done in Nutch eventually led to the creation of Hadoop as a framework for distributed storage and parallel computation (White, 2015, pp. 12–16).

With a brief introduction and background laid out, the next section explores the Hadoop ecosystem in more depth using principally a concept map along with some other supporting diagrams.

Hadoop Concept Map

Note: The diagrams/images are large but they have been inserted into this document to allow zooming without reduction in the resolution/quality.

To better organize concepts within the Hadoop ecosystem, I’ve organized the material into the following:

Core Hadoop — The core of Hadoop refers to the essential components that make Hadoop what it is. Think of the core as the foundation, or kernel, of the larger ecosystem. The functional layers around it depend on these core services for distributed storage, resource management, and parallel processing. Although the requirement of Hadoop core components is becoming less and less of a requirement as more modern tools are developed (Verbraeken et al., 2020), the core is still important to understand because it is the foundation of Hadoop and many of the ecosystem tools depend on it (Dolev et al., 2019).

Seven Functional Layers — The seven functional layers represent the major pluggable areas of the Hadoop ecosystem. These layers interact with the core Hadoop components, but they can also interact with each other through their own interfaces. This demonstrates Hadoop as an ecosystem of connected services that support storage, processing, querying, ingestion, coordination, security, governance, and analytics.

As shown in Figure 1, I made this mind map specific to a particular use case: AeroFlux. The reason for this is to try and ground all our discussion of Hadoop in the real world and not get overwhelmed with all the information that’s out there because you could easily spend your entire life learning this stuff and still not grasp it all before the next big thing comes along.

In Figure 2 I provided a layered ecosystem of Hadoop to show a sample of the available open source technologies. Most of which are supported by the same Apache Software Foundation that supports Hadoop.

Figure 3 shows a data flow diagram of the AeroFlux project in the context of a hypothetical Hadoop ecosystem. The prupose of this diagram is to support the concept map in Figure 1 by showing a more linear flow of data through the functional layers. The diagram is not meant to be a complete representation of all the possible paths through the ecosystem, but rather a sample path that shows how data could flow from ingestion to storage, processing, analytics, and governance.

Core Hadoop

The Big Four

The foundation of Hadoop is built around four core modules: Hadoop Common, HDFS, YARN, and MapReduce. Together, these components form the original foundation of Hadoop’s big data architecture and work together (Apache Hadoop Project, n.d.).

HDFS as the Foundation of Hadoop

Hadoop’s HDFS (Hadoop Distributed File System) is the main data storage mechanism in the Hadoop architecture. I call it the foundation of Hadoop. HDFS provides distributed storage by splitting large files into blocks and storing those blocks across multiple machines in a cluster. It’s data agnostic as it can handle all types of data: structured, semi-structured or unstructured data (Apache Hadoop Project, 2023).

HDFS is designed around a master-worker architecture. The NameNode acts as the master service that manages file system metadata. It keeps track of the directory structure, which blocks make up each file, and where those blocks are located across the cluster. The DataNodes are the worker machines that store the actual data blocks and serve read and write requests. When a large file is written to HDFS, it is divided into blocks, commonly 128 MB in many modern Hadoop configurations, although the block size is configurable. Each block is then distributed across DataNodes and replicated for fault tolerance. A common default replication factor is three, meaning each block is copied to three different locations so that the data remains available even if a machine fails. This design allows HDFS to store very large files reliably across clusters of commodity hardware (Apache Hadoop Project, 2023).

One of the most important ideas behind HDFS is data locality, often summarized by the Hadoop design principle that “moving computation is cheaper than moving data” (Apache Hadoop Project, 2023). The HDFS architecture guide explains that when datasets are very large, it is usually more efficient to run computation near the data instead of moving huge amounts of data across the network to a separate processing machine. This reduces network congestion and increases overall throughput. For example, if a company stores several terabytes of web server logs in HDFS, it would be inefficient to copy all of those logs to one server for analysis. Instead, Hadoop can send processing tasks to the nodes where the log blocks already exist. This is also why HDFS is optimized for streaming reads and large sequential file access rather than low-latency random access. It is very good at scanning large files from beginning to end, such as reading years of transaction logs for batch analysis, but it is not ideal when an application needs to quickly retrieve and update individual records like a traditional transactional database (Apache Hadoop Project, 2023).

Hadoop Common

Hadoop Common contains the shared libraries, utilities, and basic services that support the other Hadoop modules. It provides the Java abstractions over the underlying operating system and filesystem that HDFS, YARN, and MapReduce all build upon, along with the launch scripts, configuration handling, and JAR files needed to run a Hadoop cluster (Apache Hadoop Project, n.d.).

YARN the Manager

YARN manages cluster resources and job scheduling and monitoring. YARN completes the foundation by separating resource management from the processing model. In earlier Hadoop designs, MapReduce was tightly connected to cluster resource management. YARN made Hadoop more flexible by allowing different processing engines to share the same cluster resources. In YARN, the ResourceManager arbitrates resources across applications, while NodeManagers run on individual machines and monitor resource usage such as CPU, memory, disk, and network. This means MapReduce jobs, Spark applications, Tez jobs, and other distributed applications can run on the same Hadoop cluster while requesting resources through YARN. To better visualize these relationships consider this: HDFS stores distributed data, YARN allocates and monitors compute resources, and MapReduce uses those resources to perform parallel batch processing over the data stored in HDFS (Apache Hadoop Project, 2026a).

MapReduce the Data Processing Engine

MapReduce is the core data processing engine. A MapReduce job breaks a large task into smaller tasks that can run in parallel across the cluster. The map phase processes small portions of the input data and produces intermediate key-value pairs. The shuffle and sort phase groups all intermediate values with the same key and sends them to the appropriate reducer. The reduce phase then combines or summarizes those grouped values to produce the final output. This model gives programmers a clean abstraction for distributed processing because the framework handles task distribution, parallelization, data movement between map and reduce stages, and fault tolerance when tasks or nodes fail (Apache Hadoop Project, 2026b; Dean & Ghemawat, 2004). Although using MapReduce directly can be cumbersome, hence why why Pig, Hive, and other higher-level tools were developed to provide SQL-like interface that is more user-friendly. Those will be talked about later.

Beyond these four core components, the Hadoop ecosystem extends into additional functional layers made up of many open-source projects which is what we will explore in the following section. For a sample of what is available, see the layered Hadoop ecosystem in Figure 2.

Exploring the Hadoop Ecosystem

For the following section, we will use the concept map in Figure 1 and the AeroFlux project as a hypothetical use case. The goal is to explore the Hadoop ecosystem by tracing an end-to-end data path through the core and functional layers and showing how the different open-source projects relate to one another.

For context, AeroFlux is a proposed aviation intelligence platform that combines historical flight data, live flight streams, weather data, and airport metadata to analyze delays and operational patterns. In the context of Hadoop, AeroFlux serves as an example of how a big data architecture could ingest, store, process, query, and model large-scale aviation data for decision support. For more information on the AeroFlux project, see AeroFlux.

Seven Functional Layers Overview

Hadoop Core — The foundation of the Hadoop ecosystem. Already discussed in the previous section.

Ingestion — The data on-ramp that moves external data sources such as flight streams, weather feeds, etc. into the big data environment.

Storage — The layer that stores raw, curated, and operational data using systems such as HDFS, Parquet, HBase, MongoDB, Neo4j, and Elasticsearch.

Processing — The compute layer that transforms, cleans, enriches, and analyzes data using engines such as MapReduce, Spark, Flink, Tez, and Pig.

Data Access — The query and scripting layer that allows users and applications to interact with stored data through tools such as Hive, Trino, Pig, and Elasticsearch.

AI/ML and Analytics — The layer that uses processed and feature-engineered data for machine learning, prediction, statistical analysis, and decision support.

Security, Governance, and Operations — The cross-cutting layer that secures, manages, automates, monitors, and documents the platform through tools such as Ranger, Knox, Atlas, Kerberos, Airflow, Ansible, Ambari, and Cloudera Manager.

The AeroFlux Path Through the Layers

Data Sources enter the system.

AeroFlux begins with external aviation data sources, including flight messages, weather data, airport metadata, and historical flight records. These sources represent the raw material that the rest of the architecture is designed to capture, organize, and analyze. In a real system, examples could include FAA SWIM feeds, ADS-B or OpenSky-style aircraft position data, NOAA weather data, BTS historical flight data, and airport reference data. This part of the path sits just before the ingestion layer because these systems exist outside Hadoop but feed into the big data architecture (Bureau of Transportation Statistics, n.d.; Federal Aviation Administration, n.d.; National Weather Service, n.d.; OpenSky Network, n.d.).

[Ingestion Layer] Streaming and batch data are ingested into the platform.

AeroFlux separates ingestion into two paths: streaming ingestion and batch ingestion. For live sources such as FAA SWIM messages, AeroFlux uses Apache Kafka as the streaming backbone because the data arrives continuously and needs to be captured as events are produced. Kafka runs in KRaft mode, which means it manages Kafka metadata internally through a quorum of controller nodes instead of depending on ZooKeeper for coordination (Apache Kafka Project, 2026; Ongaro & Ousterhout, 2014). For scheduled or batch-oriented sources such as historical flight data, airport metadata, or periodic weather files, the system can use Airflow to schedule jobs that load data into HDFS or another storage layer. Other available ingestion tools in the Hadoop ecosystem include Sqoop for relational database imports, Flume for log and event collection, and NiFi for visual dataflow automation and routing (Apache Airflow Project, n.d.; Apache Flume Project, 2022; Apache NiFi Project, n.d.; Apache Sqoop Project, 2019).

[Orchestration] Workflow orchestration controls when jobs run.

This step belongs to the Security, Governance, and Operations layer, although it supports the ingestion and processing layers directly. AeroFlux uses Airflow to do more than just handle data ingestion—it sequences job tasking to run Spark jobs and trigger downstream analytics. This is different from Kafka’s internal coordination. Kafka/KRaft manages the streaming system itself, while Airflow manages pipeline workflows across systems as DAGs. In a more traditional Hadoop environment, Oozie could be used as the Hadoop-native workflow scheduler, while tools such as Luigi, Azkaban, Dagster, and Prefect are other available orchestration options (Apache Airflow Project, n.d.; Apache Kafka Project, 2026; Apache Oozie Project, 2021).

[Data Processing] Real-time events are processed before landing.

After streaming data enters Kafka, AeroFlux can use Apache Flink to process events in real time. Flink is useful for transformations such as filtering invalid messages, enriching events with airport metadata, calculating rolling airport activity, or detecting early indicators of delay propagation. This real-time processing path sits beside the batch Hadoop path but still feeds the larger data platform. Spark could also support micro-batch or structured streaming workloads, while MapReduce represents the older batch processing model. Tez and Storm are additional available processing options, with Tez commonly used under Hive and Storm representing an earlier generation of streaming technology (Apache Flink Project, n.d.; Apache Spark Project, n.d.; Apache Storm Project, n.d.; Apache Tez Project, n.d.; Dean & Ghemawat, 2004).

[Hadoop Core & Storage] Raw data lands in HDFS.

HDFS is the foundation where AeroFlux can store raw data at scale. The raw landing zone preserves data in its original or lightly transformed form so that it can be reprocessed later if the schema changes, new features are needed, or a processing error is discovered. This is one of the reasons Hadoop-style architectures are useful for data lake patterns: they allow large volumes of raw data to be stored first and interpreted later by processing and query tools (Apache Hadoop Project, 2023).

[Storage] Curated analytical data is stored in Parquet.

Once raw data is cleaned and transformed, AeroFlux stores curated analytical datasets in Parquet. Parquet is a columnar file format, which makes it useful for scan-heavy analytics where only certain columns need to be read across millions of records. For example, a delay analysis query may only need flight date, origin airport, destination airport, departure delay, arrival delay, weather conditions, and carrier. Reading only the needed columns can make analytical workloads more efficient. Other available formats include ORC, which is another columnar format often associated with Hive, and Avro, which is row-oriented and useful for record-based data exchange and schema evolution (Apache Parquet Project, 2025; Zeng et al., 2023).

[Data Processing] Batch analytics and feature engineering run on Spark.

AeroFlux uses Apache Spark for batch analytics, feature engineering, and large-scale transformations over historical and curated data. Spark can read from HDFS, process data in memory, and write results back to HDFS in formats such as Parquet. In the Hadoop ecosystem, Spark jobs can request cluster resources through YARN, while MapReduce remains the original Hadoop processing model. For AeroFlux, Spark is more useful than traditional MapReduce because it supports faster iterative analytics, machine learning workflows, and more flexible data processing (Apache Hadoop Project, 2023, 2026a; Apache Spark Project, n.d.; Zaharia et al., 2016).

[Data Access] Analysts and applications query the data through Hive and Trino.

Hive provides a SQL-like interface over data stored in HDFS and other distributed storage systems (Apache Hive Project, n.d.). It was originally developed as a SQL layer over Hadoop MapReduce, allowing users to perform ETL, batch reporting, and large-scale analysis without writing low-level MapReduce programs (Camacho-Rodríguez et al., 2019). Over time, Hive evolved into a more complete data warehousing system by improving SQL support, query optimization, and compatibility with faster engines such as Tez and Spark (Camacho-Rodríguez et al., 2019). Hive also uses the Hive Metastore to maintain information about tables, schemas, partitions, and data locations, while the underlying data can remain in HDFS, cloud object storage, or systems such as HBase (Apache HBase Project, n.d.). Trino, a relatively modern technology, adds another access path by providing fast, federated SQL across multiple data sources, which makes it useful for interactive analysis across HDFS, Hive tables, and other connected systems (Trino Software Foundation, n.d.).

[AI/ML and Analytics layer] Machine learning models predict delays.

After the data has been cleaned, joined, and feature-engineered, AeroFlux can use Spark MLlib for delay prediction. This layer connects to the processing output rather than directly to raw HDFS files, because machine learning works best when the data has already been transformed into reliable features. Mahout is an available legacy alternative in the Hadoop ecosystem, but Spark MLlib is a more natural fit for a modern Spark-based architecture (Meng et al., 2016).

[Storage and AI/ML and Analytics] propagation is modeled as a graph.

AeroFlux can use Neo4j to model network relationships i.e. propagation networks. This is useful because aviation delays are not isolated events. A graph database makes these relationships easier to explore than a flat table alone. Other graph-oriented options exist, such as JanusGraph, while HBase can support large-scale point lookups and MongoDB can store flexible document-style operational records (Apache HBase Project, n.d.; MongoDB, n.d.; Neo4j, n.d.).

[Data Access] Search and event exploration use Elasticsearch.

Here we can use Elasticsearch for fast search and exploration of flight events, logs, message contents, and operational anomalies. This is different from Hive or Trino because Elasticsearch is optimized for search-style access rather than large SQL-based analytical scans. For example, an analyst might search for all events related to a specific flight number, airport, message type, or operational disruption. Solr is an available alternative in this part of the ecosystem (Apache Solr Project, n.d.; Elastic, n.d.).

[Security, Governance, and Operations] Security, governance, and operations wrap the whole path.

Last but not lease we have security, governance, and operations. These tools do not sit in one single place in the data path because they support the entire platform. YARN allocates compute resources across jobs and applications. Ranger centralizes authorization and access policies. Knox provides a perimeter security gateway for accessing Hadoop services. Atlas manages metadata and data lineage across the stack. Kerberos handles authentication. Airflow schedules recurring workflows, while Ansible helps provision and configure machines. Ambari or Cloudera Manager could be used for cluster management and monitoring in a more traditional Hadoop environment. In the AeroFlux architecture, these supporting components make the platform manageable, secure, and explainable rather than just technically functional (Ansible Community, n.d.; Apache Airflow Project, n.d.; Apache Ambari Project, n.d.; Apache Atlas Project, n.d.; Apache Hadoop Project, 2026a; Apache Knox Project, n.d.; Apache Ranger Project, n.d.; Cloudera, n.d.; MIT Kerberos Consortium, n.d.).

References

Ansible Community. (n.d.). Ansible community documentation. Retrieved July 12, 2026, from https://docs.ansible.com/
Apache Airflow Project. (n.d.). What is airflow? Apache Software Foundation. Retrieved July 12, 2026, from https://airflow.apache.org/docs/apache-airflow/stable/index.html
Apache Ambari Project. (n.d.). Apache ambari. Apache Software Foundation. Retrieved July 12, 2026, from https://ambari.apache.org/
Apache Atlas Project. (n.d.). Apache atlas: Data governance and metadata framework for hadoop. Apache Software Foundation. Retrieved July 12, 2026, from https://atlas.apache.org/
Apache Flink Project. (n.d.). Apache flink: Stateful computations over data streams. Apache Software Foundation. Retrieved July 12, 2026, from https://flink.apache.org/
Apache Flume Project. (2022). Welcome to apache flume. Apache Software Foundation. https://flume.apache.org/
Apache Hadoop Project. (n.d.). Apache hadoop. Apache Software Foundation. Retrieved July 12, 2026, from https://hadoop.apache.org/
Apache Hadoop Project. (2023). Apache hadoop 3.3.5: HDFS architecture. Apache Software Foundation. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
Apache Hadoop Project. (2026a). Apache hadoop 3.5.0: Apache hadoop YARN. Apache Software Foundation. https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Apache Hadoop Project. (2026b). Apache hadoop 3.5.0: MapReduce tutorial. Apache Software Foundation. https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html
Apache HBase Project. (n.d.). Apache HBase reference guide. Apache Software Foundation. Retrieved July 12, 2026, from https://hbase.apache.org/book.html
Apache Hive Project. (n.d.). Apache hive. Apache Software Foundation. Retrieved July 12, 2026, from https://hive.apache.org/
Apache Kafka Project. (2026). KRaft vs ZooKeeper. Apache Software Foundation. https://kafka.apache.org/40/getting-started/zk2kraft/
Apache Knox Project. (n.d.). Apache knox gateway user guide. Apache Software Foundation. Retrieved July 12, 2026, from https://knox.apache.org/books/knox-2-1-0/user-guide.html
Apache NiFi Project. (n.d.). Apache NiFi overview. Apache Software Foundation. Retrieved July 12, 2026, from https://nifi.apache.org/docs/nifi-docs/html/overview.html
Apache Oozie Project. (2021). Apache oozie workflow scheduler for hadoop. Apache Software Foundation. https://oozie.apache.org/
Apache Parquet Project. (2025). Overview. Apache Software Foundation. https://parquet.apache.org/docs/overview/
Apache Ranger Project. (n.d.). Apache ranger introduction. Apache Software Foundation. Retrieved July 12, 2026, from https://ranger.apache.org/
Apache Solr Project. (n.d.). Welcome to apache solr. Apache Software Foundation. Retrieved July 12, 2026, from https://solr.apache.org/
Apache Spark Project. (n.d.). Apache spark. Apache Software Foundation. Retrieved July 12, 2026, from https://spark.apache.org/
Apache Sqoop Project. (2019). Apache sqoop. Apache Software Foundation. https://sqoop.apache.org/
Apache Storm Project. (n.d.). Apache storm. Apache Software Foundation. Retrieved July 12, 2026, from https://storm.apache.org/
Apache Tez Project. (n.d.). Welcome to apache tez. Apache Software Foundation. Retrieved July 12, 2026, from https://tez.apache.org/
Brin, S., & Page, L. (1998). The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems, 30(1–7), 107–117. https://research.google/pubs/the-anatomy-of-a-large-scale-hypertextual-web-search-engine/
Bureau of Transportation Statistics. (n.d.). Airline on-time performance data. U.S. Department of Transportation. Retrieved July 12, 2026, from https://transtats.bts.gov/DatabaseInfo.asp?QO_VQ=EFD
Camacho-Rodríguez, J., Chauhan, A., Gates, A., Koifman, E., O’Malley, O., Garg, V., Haindrich, Z., Shelukhin, S., Jayachandran, P., Seth, S., Jaiswal, D., Bouguerra, S., Bangarwa, N., Hariappan, S., Agarwal, A., Dere, J., Dai, D., Nair, T., Dembla, N., & Vijayaraghavan, G. (2019). Apache hive: From MapReduce to enterprise-grade big data warehousing. https://arxiv.org/abs/1903.10970
Cloudera. (n.d.). Monitoring a cluster using cloudera manager. Retrieved July 12, 2026, from https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/concepts/topics/cm-introduction-to-monitoring.html
Dean, J., & Ghemawat, S. (2004). MapReduce: Simplified data processing on large clusters. Proceedings of the 6th Symposium on Operating Systems Design and Implementation, 137–150. https://www.usenix.org/conference/osdi-04/mapreduce-simplified-data-processing-large-clusters
Dolev, S., Florissi, P., Gudes, E., Sharma, S., & Singer, I. (2019). A survey on geographically distributed big-data processing using MapReduce. IEEE Transactions on Big Data, 5(1), 60–80. https://doi.org/10.1109/TBDATA.2017.2723473
Elastic. (n.d.). Elasticsearch: The official distributed search and analytics engine. Retrieved July 12, 2026, from https://www.elastic.co/elasticsearch
Federal Aviation Administration. (n.d.). System wide information management (SWIM). Retrieved July 12, 2026, from https://www.faa.gov/air_traffic/technology/swim
Hopper, G. M. (1982). Future possibilities: Data, hardware, software, and people [Lecture transcript]. National Security Agency. https://media.defense.gov/2024/Nov/25/2003593626/-1/-1/0/PART%20ONE%20FUTURE%20POSSIBILITIES%20GRACE%20HOPPER%20TRANSCRIPT%20NO%20SUCH%20PODCAST%20NSA.PDF
Lyman, P., & Varian, H. R. (2003). How much information? 2003. University of California, Berkeley, School of Information Management; Systems. https://www.ischool.berkeley.edu/research/publications/2003/how-much-information-2003
Meng, X., Bradley, J., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Freeman, J., Tsai, D. B., Amde, M., Owen, S., Xin, D., Xin, R., Franklin, M. J., Zadeh, R., Zaharia, M., & Talwalkar, A. (2016). MLlib: Machine learning in apache spark. Journal of Machine Learning Research, 17(34), 1–7. https://www.jmlr.org/papers/v17/15-237.html
MIT Kerberos Consortium. (n.d.). Kerberos: The network authentication protocol. Massachusetts Institute of Technology. Retrieved July 12, 2026, from https://web.mit.edu/kerberos/
MongoDB. (n.d.). Data modeling in MongoDB. Retrieved July 12, 2026, from https://www.mongodb.com/docs/manual/data-modeling/
National Weather Service. (n.d.). Aviation weather center. National Oceanic; Atmospheric Administration. Retrieved July 12, 2026, from https://aviationweather.gov/
Neo4j. (n.d.). Neo4j graph intelligence platform. Retrieved July 12, 2026, from https://neo4j.com/
Ongaro, D., & Ousterhout, J. (2014). In search of an understandable consensus algorithm. Proceedings of the 2014 USENIX Annual Technical Conference, 305–319. https://www.usenix.org/conference/atc14/technical-sessions/presentation/ongaro
OpenSky Network. (n.d.). OpenSky network data. Retrieved July 12, 2026, from https://opensky-network.org/data
Reinsel, D., Gantz, J., & Rydning, J. (2018). The digitization of the world from edge to core. International Data Corporation. https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf
Trino Software Foundation. (n.d.). Trino concepts. Retrieved July 12, 2026, from https://trino.io/docs/current/overview/concepts.html
University of Sheffield. (n.d.). Clive humby. Retrieved July 12, 2026, from https://sheffield.ac.uk/cs/people/academic-visitors/clive-humby
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., & Rellermeyer, J. S. (2020). A survey on distributed machine learning. ACM Computing Surveys, 53(2). https://doi.org/10.1145/3377454
White, T. (2015). Hadoop: The definitive guide (4th ed.). O’Reilly Media.
Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., Ghodsi, A., Gonzalez, J., Shenker, S., & Stoica, I. (2016). Apache spark: A unified engine for big data processing. Communications of the ACM, 59(11), 56–65. https://doi.org/10.1145/2934664
Zeng, X., Hui, Y., Shen, J., Pavlo, A., McKinney, W., & Zhang, H. (2023). An empirical evaluation of columnar storage formats. https://arxiv.org/abs/2304.05028
]]> tech hadoop big data data architecture https://jonathanwil.com/blog/posts/tech/hadoop/hadoop-ecosystem-paper.html Sun, 12 Jul 2026 00:00:00 GMT