Big Data and Why Hadoop

In 2006, British mathematician Clive Humby coined the phrase, “Data is the new oil,” and today it seems more relevant than ever (University of Sheffield, n.d.). In 2002, about 5 exabytes of new information were stored worldwide, while IDC projected that the global datasphere would reach 175 zettabytes by 2025 (Lyman & Varian, 2003; Reinsel et al., 2018). If data is the oil, then what are the refineries? What systems can process these enormous amounts of data and turn them into useful products?

Organizations need ways to store and process data beyond the limits of a single server and many traditional databases. Instead of relying on one increasingly powerful machine, distributed systems divide the work across many computers. As Grace Hopper explained through her oxen analogy, when one ox cannot move a heavy load, you add more oxen rather than trying to build a larger one (Hopper, 1982). Hadoop follows this same idea by allowing clusters of computers to store and process data together.

What is Hadoop

Apache Hadoop is best understood as an open-source big data ecosystem rather than a single tool. It provides a framework for storing and processing large datasets across clusters of computers. Its four core modules—Hadoop Common, HDFS, YARN, and MapReduce—support distributed storage, resource management, and parallel processing, while related projects extend Hadoop into areas such as querying, analytics, governance, and data management. Unlike a traditional database, Hadoop is designed to scale from a single machine to thousands of computers working together (Apache Hadoop Project, n.d.).

Origins of Hadoop

Hadoop was born from two key developments that happened around the same time in the early 2000s, when web search and large-scale data processing were becoming increasingly difficult to manage with traditional systems. One major development was Google’s work on web-scale search. PageRank, created by Larry Page and Sergey Brin, helped make Google successful by ranking web pages based on link relationships across the web (Brin & Page, 1998). As Google grew, it also needed a better way to process massive amounts of web data across many machines, which led Jeffrey Dean and Sanjay Ghemawat to publish the influential MapReduce paper (Dean & Ghemawat, 2004). Around the same time, Doug Cutting, the creator of Apache Lucene, was working on Apache Nutch, an open-source web search project that needed scalable storage and processing. Hadoop grew out of this work as part of the Lucene/Nutch ecosystem, and its interesting name famously came from Cutting’s child’s stuffed toy elephant. The combination of Google’s MapReduce ideas, the needs of open-source web search, and the work being done in Nutch eventually led to the creation of Hadoop as a framework for distributed storage and parallel computation (White, 2015, pp. 12–16).

With a brief introduction and background laid out, the next section explores the Hadoop ecosystem in more depth using principally a concept map along with some other supporting diagrams.

Hadoop Concept Map

Note: The diagrams/images are large but they have been inserted into this document to allow zooming without reduction in the resolution/quality.

To better organize concepts within the Hadoop ecosystem, I’ve organized the material into the following:

Core Hadoop — The core of Hadoop refers to the essential components that make Hadoop what it is. Think of the core as the foundation, or kernel, of the larger ecosystem. The functional layers around it depend on these core services for distributed storage, resource management, and parallel processing. Although the requirement of Hadoop core components is becoming less and less of a requirement as more modern tools are developed (Verbraeken et al., 2020), the core is still important to understand because it is the foundation of Hadoop and many of the ecosystem tools depend on it (Dolev et al., 2019).

Seven Functional Layers — The seven functional layers represent the major pluggable areas of the Hadoop ecosystem. These layers interact with the core Hadoop components, but they can also interact with each other through their own interfaces. This demonstrates Hadoop as an ecosystem of connected services that support storage, processing, querying, ingestion, coordination, security, governance, and analytics.

As shown in Figure 1, I made this mind map specific to a particular use case: AeroFlux. The reason for this is to try and ground all our discussion of Hadoop in the real world and not get overwhelmed with all the information that’s out there because you could easily spend your entire life learning this stuff and still not grasp it all before the next big thing comes along.

In Figure 2 I provided a layered ecosystem of Hadoop to show a sample of the available open source technologies. Most of which are supported by the same Apache Software Foundation that supports Hadoop.

Figure 3 shows a data flow diagram of the AeroFlux project in the context of a hypothetical Hadoop ecosystem. The prupose of this diagram is to support the concept map in Figure 1 by showing a more linear flow of data through the functional layers. The diagram is not meant to be a complete representation of all the possible paths through the ecosystem, but rather a sample path that shows how data could flow from ingestion to storage, processing, analytics, and governance.

Core Hadoop

Exploring the Hadoop Ecosystem

For the following section, we will use the concept map in Figure 1 and the AeroFlux project as a hypothetical use case. The goal is to explore the Hadoop ecosystem by tracing an end-to-end data path through the core and functional layers and showing how the different open-source projects relate to one another.

For context, AeroFlux is a proposed aviation intelligence platform that combines historical flight data, live flight streams, weather data, and airport metadata to analyze delays and operational patterns. In the context of Hadoop, AeroFlux serves as an example of how a big data architecture could ingest, store, process, query, and model large-scale aviation data for decision support. For more information on the AeroFlux project, see AeroFlux.

References

Cusco

Our journey began in Cusco, the historic capital of the Inca Empire. The city sits at approximately 3,400 meters, or 11,150 feet, above sea level, so it also served as our introduction to the effects of high altitude.

The altitude was noticeable, but it was not as bad as I expected. I did not feel sick when we arrived, although walking uphill or climbing stairs made it immediately apparent that less oxygen was available. Even relatively simple physical activities required more effort than they would at sea level.

Cusco is a fascinating combination of Inca foundations and Spanish colonial architecture. In many places, colonial buildings and churches were constructed directly on top of older Inca stonework. Walking around the city feels like moving through several periods of history at once.

Some of the places we visited included:

• Plaza de Armas — Cusco’s central square, surrounded by colonial buildings, restaurants, shops, and historic churches.
• Cusco Cathedral — A massive colonial cathedral containing religious artwork and architecture built over earlier Inca foundations.
• Qorikancha — The former Inca Temple of the Sun, which was later incorporated into the Spanish Convent of Santo Domingo.
• Sacsayhuamán — An enormous Inca complex overlooking Cusco, known for its precisely fitted stone walls.
• Centro de Textiles Tradicionales del Cusco — A cultural center preserving traditional Andean weaving techniques and textile designs.
• San Pedro Market — A busy local market filled with produce, prepared food, fruit juices, spices, textiles, and everyday goods.

We visited several additional ruins and cultural sites throughout Cusco and the surrounding region. One of the most impressive aspects of the Inca sites was the scale and precision of their stonework. Many of the stones are enormous, irregularly shaped, and fitted together so precisely that almost no space remains between them.

The Sacred Valley

We also traveled through the Sacred Valley, visiting areas such as Pisac and Ollantaytambo. The valley contains farmland, villages, mountains, archaeological sites, and terraces that demonstrate how effectively the Incas adapted to the mountainous terrain.

Ollantaytambo was especially memorable. The town still follows portions of its original Inca street plan and sits beneath an impressive archaeological complex built into the mountainside. It was also one of the major staging points for travelers heading toward the Inca Trail and Machu Picchu.

The Salkantay Trek

The most memorable part of the trip was our five-day trek through the Andes. The route covered roughly 50 miles and combined the Salkantay route with portions of the Inca Trail.

At the beginning of the trek, the landscape was open and mountainous, with distant glaciers and snow-covered peaks surrounding the trail.

From Glaciers to Jungle

One of the most remarkable parts of the trek was how quickly the landscape changed. We began among glaciated mountains and high-altitude valleys, but after crossing the pass, we gradually descended into cloud forest and tropical rainforest.

Within a relatively short distance, the environment changed from rocky, cold, and exposed to warm, green, and humid.

We saw a surprising amount of wildlife along the way, including insects, birds, and other animals adapted to the different elevations. One of the more unusual sightings was a high-altitude tarantula that our guide identified along the trail.

We also passed farms growing coffee, cacao, passion fruit, sugar cane, and other tropical crops. Being able to see and taste food directly where it was grown made this portion of the trip especially memorable.

Coffee and Chocolate

During the lower-elevation portion of the trek, we participated in a coffee-making experience. We started with raw coffee beans and worked through several of the steps involved in preparing them.

We also tasted raw cacao beans and learned more about how cacao is transformed into chocolate. Raw cacao tastes very different from the finished chocolate most people are familiar with, making it interesting to experience the ingredient closer to its original form.

Life on the Trail

The trekking company handled nearly all of the logistics. Our group had a guide, two chefs, and mules that carried much of the equipment between camps. At night, we stayed in a mixture of small lodges and glamping-style accommodations rather than carrying and setting up traditional tents ourselves.

The level of support made the trip considerably more comfortable than I originally expected. Meals were prepared for us each day, transportation was coordinated, and our guide explained the history, geography, plants, and cultural significance of the places we encountered.

Our guide also regularly chewed coca leaves. Coca has a long history in Andean culture and is commonly used by people living and working at high elevations. The leaves are also used in tea and are believed to help people manage fatigue and some of the effects of altitude.

There was a running joke that the Incas managed to build their enormous mountain complexes with a little help from coca leaves. Joking aside, the plant clearly has a deep cultural significance that goes far beyond its association with cocaine.

The Inca Trail

As we moved closer to Machu Picchu, portions of the route joined historic Inca trails. Hiking these sections provided a better appreciation for the scale of the Inca road system and the difficulty of building paths through such steep terrain.

At several points, we could see Machu Picchu in the distance before reaching the site itself. After several days of hiking, seeing it appear across the mountains made the destination feel much more significant than arriving by train or bus alone.

Aguas Calientes

Before entering Machu Picchu, we spent time in Aguas Calientes, the primary town serving visitors to the archaeological site. The town sits in a narrow valley surrounded by steep, forested mountains, with a river and railway running through it.

The train system connecting the region also provides some incredible views as it winds between the mountains and follows the river toward Aguas Calientes.

Machu Picchu

Machu Picchu was the culmination of the trek and the place I had most anticipated seeing. Photographs capture its basic shape, but they do not fully communicate how dramatically the site is positioned within the surrounding mountains.

The combination of architecture, elevation, terraces, clouds, and steep green peaks makes the entire setting feel almost unreal.

Walking through the site also makes it easier to appreciate its complexity. Machu Picchu was not simply a collection of stone buildings. It incorporated agricultural terraces, water systems, residences, ceremonial spaces, stairways, and carefully planned pathways throughout a steep mountain environment.

After hiking for several days, reaching Machu Picchu on foot made the experience feel like the conclusion of a much larger journey rather than a single sightseeing stop.

Rainbow Mountain and the Red Valley

Another major excursion was Rainbow Mountain and the nearby Red Valley. Rainbow Mountain reaches approximately 5,036 meters, or 16,522 feet, making it one of the highest elevations we reached during the trip.

The altitude was again the greatest challenge. Even after spending several days in the mountains, walking uphill at more than 16,000 feet required a slow and deliberate pace.

The neighboring Red Valley was equally impressive and, in some ways, even more striking. The landscape contained broad red slopes and ridgelines that looked almost like another planet.

Food in Peru

Food was another major part of the experience. We tried a wide range of dishes and ingredients, including:

• Alpaca
• Guinea pig, traditionally known as cuy
• Several varieties of ceviche
• Fresh tropical fruit
• Raw cacao beans
• Locally grown coffee
• Sugar cane
• Inka Cola
• Coca tea and coca leaves

The ceviche in Lima was some of the best I have ever had. It was extremely fresh, bright, acidic, and very different from many versions I had tried previously.

Guinea pig was one of the most unusual foods we sampled. It had a flavor and texture unlike anything I had eaten before. It was not necessarily something I would order regularly, but I am glad I tried it as part of the cultural experience.

San Pedro Market in Cusco was one of the best places to explore local food at an affordable price. The market offered fruit, juices, soups, meats, baked goods, spices, and prepared meals alongside textiles and household products.

Inti Raymi and Andean Culture

We also experienced some of the celebrations surrounding Inti Raymi, or the Festival of the Sun, which takes place every June. The festival celebrates the Inca sun god and remains one of Cusco’s most important annual cultural events.

Throughout the trip, we encountered traditions connected to weaving, farming, food, music, clothing, and religious practice. Visiting the textile center and seeing how alpaca wool is processed and woven helped illustrate the amount of skill involved in creating traditional Andean clothing.

We also had opportunities to feed alpacas and learn more about how alpaca fiber is cleaned, dyed, spun, and turned into clothing and textiles.

Trekking Company

For anyone interested in completing a similar trip, we used Salkantay Trekking.

The company was professional and well organized, especially compared with some of the other trekking operations we observed along the route. Transportation, accommodations, food, entrance arrangements, and guides were coordinated in advance, which allowed us to focus on the experience itself.

Our guide was knowledgeable about the trail, Inca history, local culture, wildlife, and plant life. The chefs also managed to prepare surprisingly elaborate meals despite working from temporary kitchens along the trail.

The company offers several different packages and routes. Our experience included glamping-style accommodations, while some versions of the traditional Inca Trail involve more basic backcountry camping conditions. I originally expected our accommodations to be much more rugged, so the level of comfort was a pleasant surprise.

Final Thoughts

Peru was one of the most varied trips I have taken. Within a relatively small geographic area, we moved from colonial cities and ancient ruins to glaciers, high-altitude mountain passes, cloud forests, tropical farms, and dense jungle.

The altitude, illness, and long days of hiking made portions of the trip challenging, but those difficulties also made the experience more memorable. Reaching Salkantay Pass, seeing Machu Picchu emerge in the distance, preparing coffee from locally grown beans, and walking through landscapes that changed almost daily were all experiences I will remember for a long time.

Machu Picchu may have been the destination, but the journey through Peru was what made the trip truly meaningful.